Engineering.Reference:26.4.1.5 - Change the learned objective deliberately
A policy may first need to learn the interaction protocol and later to solve the task. Label the stage and its objective when preparing feedback. R1-Searcher v2, §§2–3 first gives 0.5 for using retrieval at least once and 0.5 for acceptable format, with no answer reward. Its second stage removes the retrieval bonus, uses answer F1 and penalizes format violations by −2. This is a change in what receives reinforcement. The later policy still needs retrieval and its external evidence; a protocol reward has not taught all the reasoning required for the answer.
In the calculator example, all four episodes use a call and a well-formed answer. Giving each reward 1 for those properties produces zero group-relative update. Giving only C reward 1 makes addition more likely. Both calculations can be correct while their feedback fails the intended task. Preserve the original observations and identify the objective used to rescore or collect each batch. Decide the transition using actual usable interaction and later task behavior. A stage number alone supplies no evidence that the earlier capability was acquired.
Check the reward against the behavior it could make attractive. Tool-Star’s collaboration bonus uses search and Python markers in a correct response; marker presence alone does not establish that either tool contributed. R1-Searcher’s reported failures include fabricated document spans and long answers that collect likely gold-answer words. Bind tool observations to execution, check the required result independently and test plausible reward exploits. Redesign or narrow a defeated signal before extending reliance on its optimized value.
Finer intermediate feedback is another construction with its own costs. The 2026 outcome/process comparison, §§3–4 and Limitations finds that missing retrievable evidence can change the comparison; terminal-reward training remains competitive in its tested settings, while process signals trade query quality, redundancy and depth. First check that the receiving environment can supply the needed evidence and that the outcome baseline is usable. More detailed reward cannot retrieve a fact absent from the available corpus.
If long interactions still leave an important attribution problem, TRACE v1, §§2–3 and Appendix A supplies a progress-estimation alternative. A frozen reference model scores the known answer at successive tool-boundary prefixes; changes in log likelihood contribute turn-level credit combined with outcome advantage. That route needs reference answers, prefix-scoring access and a qualified estimator. Increased answer predictability is the estimator’s signal; it does not by itself establish causal usefulness of a call. Qualify it for the receiving outputs and preserve its aggregation with terminal feedback. A missing reference answer or unreliable scorer returns that branch to obtaining feedback, while a viable outcome method can remain usable.
The resulting engineering choice is between available, qualified learning constructions. Start with the signal and support available for the work; obtain a missing prerequisite when the needed distinction depends on it. Carry the selected update into subsequent execution, then compare actual task results, preserved useful behavior and total cost on appropriate separate cases. A larger reward, more prepared examples or a changed parameter value alone does not establish the intended improvement.