Post-training a 4B retail agent with hard-token trajectory SFT

TL;DR

I post-trained Qwen3.5-4B to operate a retail support agent on Tau2-bench. One epoch of task-matched hard-token trajectory SFT moved the best student from 81.88% to 85.63% Pass^1 on a 40-task, four-trial selection grid. That is a 3.75-point gain, but the paired bootstrap interval still includes zero. The result is promising, not conclusive.

The most useful findings were not the headline score. A second candidate reached lower training loss and performed worse than the base model. A later GEPA prompt search gained 5.41 points on its 74 training tasks and lost ground on the 40-task grid. End-to-end agent evaluation, rather than loss or prompt-search fitness, was the only reliable selector.

The task is a conversation that changes state

Tau2 retail is not a set of isolated questions. A simulated customer asks for help with an order, address, return, exchange, cancellation, or payment. The agent must authenticate the customer, inspect records through tools, follow policy, obtain confirmation before a write, perform the correct database action, and communicate the required facts.

A trajectory can look helpful and still fail. The database grader checks whether the world ended in the correct state. A natural-language grader checks whether the agent told the customer what the task required. The attempt receives full reward only when both are correct.

The agent harness was one PydanticAI model in a ReAct loop. It was not a workflow of several LLMs. The same model alternated between reasoning, speaking to the simulated customer, and calling one of Tau's typed tools. Tau executed each tool and returned the observation to that same agent.

DeepSeek V4 Flash played three separate roles in the experiment. It was the teacher that generated successful training trajectories, the customer simulator used during evaluation, and the natural-language judge. Only verified teacher or student agent actions became training targets. Simulator replies and judge outputs never did.

That distinction matters. This was one deployable student model with tool use, not several models trained to imitate a staged workflow.

Why 74, 40, and 114 all appear

The official retail corpus contains 114 tasks. Upstream provides 74 train tasks and 40 disjoint test tasks. Its leaderboard base view is the exact union of those sets.

I deliberately used all 74 training tasks for data and selected directly on the 40 test tasks. There was no separate development set and no untouched final student evaluation. That choice gives more task coverage for a small study, but it also means every student number in this post is selection-aware. It must not be read as a clean estimate on unseen tasks.

The 114-task teacher run is different. I ran it after the harness was frozen to build a complete custom submission. Its score is not directly comparable to the student's 40-task result because it covers a different task set. The distinction is inconvenient, but merging those numbers would be worse.

The exact benchmark source was Tau2-bench v1.0.1 at commit fc0055dc4e0a316c3f83133267fbd6faaa770992. The Qwen parent was revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. The ReAct v1 system prompt was frozen at SHA-256 4fac591d3cd8f77913afafde5bfd3b841f6822f818b6ab450abc7c9abd78cce5.

The data was successful behavior plus corrections at student states

I first ran DeepSeek and the untrained Qwen agent twice over every one of the 74 training tasks. Successful DeepSeek trajectories supplied strong demonstrations. Successful Qwen trajectories supplied retention data, so training would not contain only behavior from another policy.

For failed Qwen attempts, I asked the teacher to continue from the student's exact conversation and database state. This is the part that changes the data distribution. A clean teacher demonstration shows how a strong model behaves from the original task. It does not show how to recover after the student has asked an awkward question, selected an unhelpful tool, or navigated into a state the teacher would never create on its own.

The recovery collector tried 29 failed Qwen attempts and found verified continuations for 21 of them across 18 tasks. Every continuation had to pass the same environment reward. Fluency was not a gate.

The final Hugging Face Dataset contained 2,357 turn-target rows across 72 of the 74 training tasks:

Tasks 105 and 110 were attempted but produced no verified successful retained or recovered trajectory. I excluded them instead of training on an answer that the environment had marked wrong.

The rows ranged from 5,265 to 17,002 tokens, with a median of 7,653 and a 95th percentile of 13,095. The training context was 32,768 tokens, so no row was truncated.

The exact method was SFT

The method was fixed-target hard-token teacher-trace SFT with student-state hard correction and Qwen success retention. It can also be described as offline hard-token distillation because a stronger model produced many of the targets. It was not soft knowledge distillation, on-policy distillation, DPO, or reinforcement learning.

For each target turn, the input contained the policy, tool definitions, and preceding conversation state. The labels masked the context and scored only the target agent tokens. In simplified form:

prompt = render(system_prompt, tools, conversation_state)
target = render_agent_turn(tool_call_or_response)
input_ids = tokenize(prompt + target)
labels = [-100] * len(tokenize(prompt)) + tokenize(target)

This teaches one decision at a time while retaining the state that made the decision meaningful. Tool calls remain native tool calls. They are not converted into prose or into a separate planner schema.

I trained two LoRA candidates with the same data and recipe. Only the learning rate changed.

Candidate A trained for 3.805 hours. Candidate B trained for 3.926 hours. Both completed all 295 steps, saved valid adapters, reloaded through vanilla PEFT, and had 128 nonzero LoRA-B tensors before and after reload. I also served each adapter through vLLM and ran the real ReAct path before accepting its evaluation.

A falling loss was not enough to trust an adapter

The save-and-reload checks were not ceremony. A LoRA run can finish with a plausible loss curve and still leave behind an unusable adapter, a zeroed tensor set, or a checkpoint that only reloads through the exact training process that created it. Any of those failures would make a later evaluation look like an optimization problem when it was actually an artifact problem.

The smoke test therefore covered the complete path before the paid run: construct one real training row, run a short update, save the adapter, reload it through PEFT in a fresh model process, verify that LoRA-B tensors changed, serve it through vLLM, and complete one scored Tau trajectory. After full training, the same checks ran again for both candidates.

The evaluation artifacts were also atomic and resumable. Each task and seed wrote one terminal attempt file. A run manifest could say running, completed, or failed, and only completed counted. This mattered when an unstable Wi-Fi tunnel interrupted later GEPA evaluations. The remote model server stayed healthy, completed attempts remained immutable, and the runner resumed only the missing task-seed paths. A file existing was never treated as proof that its run had finished.

These checks do not improve a model. They make sure that a model score belongs to the weights, prompt, tasks, and trial seeds it claims to represent.

One epoch moved the best student toward the teacher

The full four-trial result is below. Pass^1 is the mean success probability of one sampled run per task. Higher powers reward consistency: Pass^4 estimates the probability that four sampled runs all succeed.

Candidate A gained 3.75 Pass^1 points over the base and closed about 35% of the measured base-to-teacher gap on this grid. Its Pass^4 gain was larger at 7.50 points, which suggests the adapted policy was more consistent across repeated conversations.

The score is not enough to establish a gain. One test task lacked reference behavior represented in the training distribution, so the preregistered selector used the matched set of 39 tasks. There Candidate A gained 3.85 points over the base. A paired task bootstrap with 100,000 resamples produced a 95% interval from -3.85 to +11.54. Zero is inside the interval.

That is why I describe the outcome as a positive selection result, not proof that this recipe reliably improves a new sample from the distribution.

Why I ran four trials per task

The customer simulator and agent are stochastic. A task that passes once can fail when the customer phrases a reply differently or when the agent samples another valid-looking action. A one-trial result would collapse that variation into a brittle binary score.

Four trials make two things visible. Pass^1 estimates average one-shot success over all 160 attempts. Pass^4 falls quickly when a task is only sometimes solved, so it highlights consistency. Candidate A's larger Pass^4 gain is encouraging, but four trials still do not make 40 tasks a large sample.

The uncertainty calculation kept each task as the resampling unit. It resampled paired base and student task vectors rather than treating 160 attempts as independent examples. That preserves the fact that four attempts on one underlying task share the same policy constraints and database goal. Treating every attempt as independent would produce an interval that looks more confident than the experiment warrants.

The net gain hid both learning and forgetting

Across the same 40 tasks, Candidate A improved nine tasks, regressed on six, and tied the base on 25. At the individual attempt level it won 17 times, lost 11 times, and tied 132 times.

This is the most useful way to read a 3.75-point net change. The model did not acquire a clean set of new skills while preserving everything it knew. The data shifted probability mass in both directions. Retention examples reduced forgetting, but did not remove it.

A concrete case shows the kind of difference that mattered. In task 49, the customer wanted to exchange IPX7 earbuds for the cheapest other earbuds already present in the same delivered order. The base model correctly described the intended item in its confirmation, then sent item ID 8555936349, a cheaper catalog variant that was not the requested item from the existing order. Its natural-language score passed and its database score failed.

Candidate A selected item ID 1646531091, the cheapest qualifying earbuds already in that order, and passed both graders. The improvement was not a nicer response. It was a more faithful mapping from conversation constraints and retrieved records to the tool argument that changed the world.

Lower loss selected the worse model

Candidate B ended with lower training loss: 0.295868 compared with Candidate A's 0.305301. It also scored 80.00% Pass^1, below both Candidate A and the 81.88% base.

This was the cleanest negative result in the study because almost everything else was held constant. The same 2,357 rows, parent revision, one epoch, batch, LoRA shape, optimizer, schedule, seed, context length, save path, serving stack, harness, tasks, and trial seeds were used. Increasing the learning rate from 5e-5 to 1e-4 improved next-token fit to the training rows and damaged end-to-end behavior.

Training loss answers whether the adapter predicts the fixed targets it saw. The task metric asks whether a stochastic multi-turn policy can authenticate, retrieve the right records, respect policy, choose valid tools, preserve IDs across turns, write the correct state, and communicate the right facts to a new simulated customer. Those are related objectives, but they are not interchangeable.

If I had selected by loss, I would have promoted the worst student.

Prompt optimization improved train and failed the same test

After SFT, I tried GEPA on the harness. The optimization surface was intentionally narrow: one generic instruction suffix for the same single PydanticAI ReAct agent. There was no additional planner, critic, reviewer, or model call in the deployed harness.

GEPA evaluated Candidate A on all 74 training tasks at one seed. The seed suffix scored 58/74, or 78.38%. Its selected proposal scored 62/74, or 83.78%. That is a gain of four tasks and 5.41 points on the data used by the optimizer.

I froze the selected suffix before running either model on the 40-task grid. The result went in the other direction.

The 74-task improvement was resubstitution performance. It did not generalize to the disjoint 40-task grid. Candidate A under the original ReAct v1 instruction remains the best measured student.

GEPA did what the optimization objective asked. The mistake would have been treating a better train fitness value as proof of a better harness. This is the same lesson as the loss comparison in another form: an internal objective becomes trustworthy only after the full task metric confirms it on data the search did not use.

The search itself was inexpensive relative to training. It used 228 logical metric calls, 222 unique completed task attempts, and two DeepSeek reflection calls. Tracked DeepSeek simulation, judging, and reflection cost $0.072586. The negative result was worth keeping because it prevented a prompt selected on train from silently replacing the better frozen harness.

What the measured work cost

The two SFT runs consumed 7.731 GPU hours in total. Candidate A took 3.805 hours and Candidate B took 3.926. The complete 160-attempt base grid took 2.319 hours. Serving both trained candidates and completing their 320 attempts took about 2.6 hours because the student trajectories were shorter than the base timing had suggested.

I did not preserve a trustworthy single euro total for the rental, so I will not reconstruct one from a current GPU price. The measured wall times are the reproducible evidence. Hardware rates change, and this run also included setup, smoke tests, model downloads, serving restarts, and transport recovery that a bare training-time multiplication would omit.

The API ledger is more exact. On the 40-task ReAct v1 grids, tracked DeepSeek usage cost $0.338294 for the teacher, $0.073772 for the Qwen base simulator and judge, $0.071188 for Candidate A, and $0.071898 for Candidate B. The Qwen rows exclude local GPU inference because it was not metered per token. The GEPA search cost another $0.072586 in DeepSeek simulation, judging, and reflection. The complete 114-task teacher run cost $1.008280 including agent, simulator, and judge usage.

Those numbers are experiment costs, not a production cost comparison. They include simulated customers and graders that a deployed support agent would not call. The relevant operational comparison would need workload arrival rate, batching, latency requirements, GPU utilization, and a real human-escalation cost. This study did not measure those.

The complete teacher run is the leaderboard artifact

The official leaderboard accepts single-domain results, but a retail submission must cover all 114 base tasks. Modified prompts, modified scaffolds, or domain-specific training belong in the Custom category. That means neither the 40-task base score nor the 40-task student score is a leaderboard result.

I ran the frozen DeepSeek teacher harness four times over all 114 tasks and exported the stored trajectories into Tau's canonical Results format. The official public validator reproduced the following metrics from 456 complete attempts:

The agent-only average API cost recorded in the submission is $0.001756 per trajectory. The complete evaluation, including the DeepSeek simulator and natural-language judge, cost $1.008280 in tracked API usage.

The run is a Custom submission because the harness adds a modified general instruction suffix and uses DeepSeek V4 Flash as the customer simulator and judge. I disclosed those changes rather than presenting the score as a Standard result. I also published the exact submission metadata and 456 trajectories on Hugging Face, with checksums and an AI-generated-output disclosure.

The submission is pending upstream review. Until it is accepted, 89.47% is a validated custom run, not an official rank. The best student remains ineligible because I have not spent the compute to evaluate it on the complete 114-task protocol.

What the experiment actually shows

It shows that task-matched trajectory SFT can move a 4B tool-using policy toward a much stronger teacher on this bounded retail workload. The strongest evidence is not merely the aggregate gain. Candidate A made more correct database changes than the base, including states where an almost-right choice of item ID was still wrong.

It also shows how fragile the conclusion is. Nine tasks improved and six regressed. The confidence interval includes zero. The selection set was used repeatedly and was not an untouched final. A lower training loss picked the wrong adapter. A higher prompt-search fitness picked the wrong instruction. Every one of those facts narrows the claim.

It does not show that Qwen3.5-4B is generally comparable to DeepSeek V4 Flash. It does not establish that Candidate A would gain 3.75 points on a fresh sample. It does not put the student on the Tau leaderboard. It does not show that another epoch or a larger learning rate would close the remaining gap.

What I would change in the next run

The next useful experiment is not another blind training round. I would first create a larger task-matched sample or a new held-out set from the same generation process, then freeze it before selecting data recipes, learning rates, checkpoints, or prompts. The current 40 tasks have already answered too many questions to serve as a clean final.

I would keep the parts that survived this study:

• one real ReAct agent rather than a disguised multi-model workflow;
• native tool calls and exact conversation state in the training rows;
• verified teacher traces rather than fluent but unchecked continuations;
• success retention beside teacher demonstrations and student-state corrections;
• at least two learning rates, both scored end to end;
• paired task analysis and uncertainty beside every small delta;
• canonical leaderboard export only after the complete protocol is covered.

I would not select by loss or by train-only prompt fitness. Those signals remain useful for diagnosing whether optimization moved, but neither is a proxy for completing the job.

The practical takeaway

Small-model specialization is often described as a training problem. In this experiment, the training code was the simple part. The hard part was preserving the product boundary: one model, the real tools, the real policy, the real environment state, and a metric that only pays when the conversation and database both finish correctly.

That boundary caught the wrong item ID, the lower-loss regression, and the prompt that overfit. Without it, all three could have looked like progress.

The best 4B student is not a leaderboard winner and the measured gain is not yet statistically settled. It is still a useful result: one epoch, 2,357 verified targets, and a compact ReAct policy that moved in the intended direction while exposing exactly what evidence the next study needs.

If you have a repeated production agent workload and a way to verify whether it completed the job, that is the kind of boundary I want to work with. You can book 20 minutes or email me at isaac@nablo.io.

References and artifacts

• Tau2-bench repository
• Tau2 leaderboard
• Official leaderboard submission guide
• DeepSeek teacher submission metadata and trajectories
• Qwen3.5-4B model card
• GEPA repository

The teacher-generated trajectories and publication artifact are AI-generated and may contain errors. No customer data appears in this experiment.