Engineering.Reference:26.4.1.4 - Supply preparation and preference inputs when those branches are selected
When the starting policy rarely produces a usable tool interaction, demonstrations can make subsequent exploration feasible. Obtain successful executions with available task inputs, actual observations and checked outcomes. From A, construct a supervised pair from the initial history to the multiplication call, and another from the resulting history including 323 to answer 328. Fit the generated actions; supply the return in the later input. A corrected first call in C would need its own execution or other qualified consequence evidence. A label for that correction may be usable before such evidence exists, but it does not inherit C’s return 36. The correction of B’s final answer can instead use the already available 323 if the stated method warrants it.
ReTool, ICLR 2026, §2 and Appendix A develops a preparation-to-update route: produce tool-interleaved traces from checked math solutions, retain their reasoning and failed attempts, filter format and final answers, and perform cold-start supervised training before PPO on new executed trajectories. Its main reward checks final correctness rather than code executability; interpreter output remains context and is masked from loss. For an adaptation, choose one executable parser contract and exercise it on the prepared examples. Figure 9 contains conflicting instructions about Markdown fences inside code tags. A receiving choice such as raw code inside those tags, with no Markdown fences, resolves that conflict locally; it cannot be attributed to an unambiguous source instruction. If checked demonstrations or an executable serialization are unavailable, report which preparation input is missing before claiming the policy was prepared.
A preference update requires two available, comparable outputs and a justified ordering. For the same request, A and B provide a simple qualified pair: prefer the checked correct answer over the checked incorrect one. They share the same first call and observation. Retain both histories, their generated actions, the ordering grounds and a fixed reference policy. If sampling supplies only rejected outputs, obtain a warranted preferred one or use a different available feedback regime. Relabeling one of the failures as preferred would teach a different objective.
For a concrete finite preference update, let P(A) and P(B) be products of policy probabilities of the generated decisions, conditional on each recorded history. The tool return has no generated-output probability in these products. With reference probabilities from the old zero-logit policy, minimize:
loss = -log(sigmoid(beta * (
(log P_current(A) - log P_current(B))
- (log P_reference(A) - log P_reference(B)))))
Set beta to 0.3. At the reference policy, gradient descent with rate 0.1 changes only z2, from 0 to 0.015, giving unshifted-answer probability 0.503750. The shared call cancels from the pair comparison, so z1 remains zero. On the new (12,13,5) input, draws (0.49,0.501) select multiplication in both policies, then answer 162 before this preference update and 161 after it. The pair therefore distinguishes the answers but supplies no preference between call choices.
Tool-Star v1, §3.2, Algorithm 1 and Appendices B–E combines preparation, masked group-relative training and preference updates. It separates direct successes, tool-assisted successes and intended harder failed questions, then samples candidates after GRPO and uses scores at least 1 for preferred outputs and below 1 for rejected outputs. Keep the contrasting candidates and the executable reward grounds. The source’s earlier correct-trace filter leaves the supply of both-failed questions underspecified. This application keeps their question IDs before filtering. The source’s printed negative DPO loss and ascent wording also conflict. The finite example above uses gradient descent on that loss. Reversing the sign would reduce the preferred answer’s probability to 0.496250. The printed instructions leave the behavior of the authors’ implementation unresolved.