Retain useful external resources and spend effort on the missing contribution
A deployed arrangement may keep a substantial resource after learning. TInR, ACL 2026, §3 first selects learned tool tokens, retrieves the selected tools’ current descriptions and parameters, then generates arguments. Preserve that document lookup with the executor. SkillRL v1, §3 trains a policy with a retrieved library and expands it from failures. SkillForge v1, §3 makes requesting guidance an action and later revises guidance from experience. Return content before the action that needs it: guidance delivered alongside an already executed action can inform only its continuation. SYSE.48 constructs the usable resource, .52 exposes it and .47 consumes the return. Separate bad retrieval, ignored guidance and a faulty instruction before deciding which to repair.
Additional reasoning is another external contribution with its own cost. HeavySkill v1, §§2–3 compares candidate trajectories through deliberation; ParallelMuse v1, §3 branches from selected prefixes and aggregates compressed reports. When choosing such a procedure, name the available branching signal, reused prefix, retained observations and aggregation rule. Preserve a consequential disagreement and a path back to its source through compression. Count initial generation, branches, tools, summaries, synthesis and completion. Shared prefixes or sources make agreement dependent.
In the range example, several branches using revision 7 can agree that 15 seconds is acceptable while the required revision is 8. Another such branch cannot supply the current rule. Spend the next effort on that observation, or return the unavailable source. Fishing for Answers v1, §3 and Appendices B–C supplies useful retrieval choices while showing that selecting, cropping and deleting context can interact: a combination can discard what the next action needs. Preserve the original question and jointly necessary premises when reformulating or shortening. Once the current revision and correctly converted amount settle the report, finish.
Learning a shorter or less-assisted route also requires a qualified result criterion. Aware First, AAAI 2026, §3 uses model-relative sampled correctness and length to construct training groups. EAPO v2, §3 contrasts tool-free and tool-available attempts and changes rewards and selected token weights from sampled correctness and calls. Those are obtainable training statistics; a sampled minimum call count is not proof that a current observation is unnecessary. For this report, a memorized but accidentally correct range answer lacks the required current-revision evidence and is not a successful training outcome. Apply a call-cost preference to a correctly specified result and preserve its validity conditions.
To calculate such a signal, hold conversion correct and vary only the next effort choice. Three available routes make one current-rule read, make two reads, or finish from the information already available. Initially no current rule is supplied. Stipulate one recorded attempt per route from a uniform policy over these three routes; the first two obtain the same current rule and produce valid reports, while the third uses a remembered range. Let q=1 for a correct report supported by the required current-revision evidence and q=0 otherwise. With c additional reads in {0,1,2}, choose the receiving reward r=q*(1-0.1*c). This explicitly protects validity before rewarding lower call cost.
| Recorded route | Additional reads c | Valid result q | Reward r |
|---|---|---|---|
| Read once and report | 1 | 1 | 0.9 |
| Read twice and report | 2 | 1 | 0.8 |
| Finish from remembered information | 0 | 0 | 0.0 |
Reuse 26.4.1’s clipped-ratio objective with one generated route choice per attempt, old route probabilities (1/3,1/3,1/3), frozen population-standardized advantages, clipping interval [0.8,1.2] and ascent step 0.3. Parameterize the route probabilities by p_i=exp(z_i)/sum_j exp(z_j), initially z=(0,0,0). The reward mean is 0.566667 and population standard deviation 0.402768, giving advantages (0.827606, 0.579324, -1.406930). At this initial policy every ratio is one. Since the three advantages sum to zero, the mean surrogate gradient is dJ/dz_i=A_i/3. One step gives logits (0.082761, 0.057932, -0.140693) and route probabilities (0.360331, 0.351494, 0.288175). A later route draw of 0.35, in the table’s order, changes the choice from two reads under the old policy to one under the changed policy; the retained reader then supplies the actual rule for the report.
Now supply a qualified current snapshot before the effort choice. Finishing from that available information yields a valid report with zero additional reads. A separate comparison from the same uniform policy has rewards (0.9,0.8,1.0) and advantages (0.000000, -1.224745, 1.224745). The same first-step construction gives probabilities (0.331673, 0.293440, 0.374887), making finish the most probable route. The two calculations qualify different input conditions; retain that condition in the policy’s input and use separate logits for these two finite states. Reusing the first state’s failure label would penalize a sufficient answer. For further steps, evaluate the clipped objective with the stored old probabilities and frozen advantages, as 26.4.1 requires.
A direct rule can make one read when current evidence is missing and finish when it is supplied. The following cost comparison uses that rule. If an effort learner is selected for a broader task, include its attempt collection, fitting and assessment in the whole cost.