Engineering.Reference:26.4.1.2 - Calculate an outcome update and consume it
Choose a mean of separate generated-decision terms. Let A_i be episode i’s reward minus the group mean, divided by the population standard deviation. Use zero advantages for a zero-deviation group. For this group the advantages are (sqrt(3), -1/sqrt(3), -1/sqrt(3), -1/sqrt(3)). Each episode has two generated decisions. A complete surrogate for this finite adaptation is:
rho_it = current_probability(action_it | history_it)
/ old_probability(action_it | history_it)
J = mean_over_episodes[
mean_over_generated_decisions[
min(rho_it*A_i, clip(rho_it, 0.8, 1.2)*A_i)
]
]
Maximize J. This example uses no reference-policy regularizer. Keep the old probabilities fixed during this update. At the old policy all ratios are 1 and clipping is inactive. For either logit, the derivative of a chosen binary action’s log probability is choice_bit - 0.5, where bit 1 denotes multiplication or o+k. Averaging both decisions and all four episodes gives gradient (1/(4*sqrt(3)), 1/(4*sqrt(3))). An ascent step of size 0.3 gives logits (0.0433012702, 0.0433012702) and choice probabilities (0.5108236264, 0.5108236264).
This code prepares the histories, obtains the first update and uses the resulting probabilities on a new input.
from math import exp, sqrt
def sigmoid(z):
return 1 / (1 + exp(-z))
def calculator(n, m, multiply):
return n * m if multiply else n + m
# Recorded choices, returns and answers; the old logits are (0, 0).
raw = [(1, 1, 323, 328), (1, 0, 323, 329),
(0, 1, 36, 41), (0, 0, 36, 42)]
request = (17, 19, 5, "calculator evaluates the selected expression exactly")
batch = []
for call_bit, answer_bit, observed, answer in raw:
n, m, k, tool_meaning = request
expression = f"{n}*{m}" if call_bit else f"{n}+{m}"
assert observed == calculator(n, m, call_bit)
assert answer == observed + k + (1 - answer_bit)
batch.append({
"generated": [
{"history": request, "action": ("call", expression),
"bit": call_bit, "old_probability": 0.5},
{"history": (request, ("call", expression), ("return", observed)),
"action": ("answer", answer), "bit": answer_bit,
"old_probability": 0.5}],
"reward": int(answer == n * m + k)})
def first_step(records, rewards, rate=0.3):
# Only this frozen zero-logit policy's first step; two selected positions.
count = len(records)
mean = sum(rewards) / count
sd = sqrt(sum((r - mean)**2 for r in rewards) / count)
advantages = [(r - mean) / sd for r in rewards] if sd else [0.0] * count
gradient = [sum(advantages[i] * (records[i]["generated"][j]["bit"] - 0.5)
/ 2 for i in range(count)) / count for j in range(2)]
return [rate * g for g in gradient]
def act(probabilities, n, m, k, draws):
multiply = draws[0] < probabilities[0]
observed = calculator(n, m, multiply)
unshifted = draws[1] < probabilities[1]
return observed + k + (0 if unshifted else 1)
old = [0.5, 0.5]
logits = first_step(batch, [row["reward"] for row in batch])
new = [sigmoid(z) for z in logits]
assert act(old, 12, 13, 5, (0.51, 0.51)) == 31
assert act(new, 12, 13, 5, (0.51, 0.51)) == 161
assert first_step(batch, [1, 1, 1, 1]) == [0.0, 0.0]
wrong = [sigmoid(z) for z in first_step(batch, [0, 0, 1, 0])]
assert wrong[0] < 0.5 # Rewarding C would encourage addition.
print([round(p, 10) for p in new])
Both runs retain the calculator. Reset the task’s history for the new input; retain the updated logits. Under these supplied choices, the probability of selecting multiplication and the unshifted answer rises from 0.25 to approximately 0.260941. The fixed draws were chosen to show one changed action. That probability is exact for the stated task and controller. An estimate for other tasks needs a justified model of those tasks or observations covering the population of interest. Use an independent evaluation sample when assessing performance empirically after fitting the policy. The representation already makes the same choices applicable to new n, m and k. The calculation establishes a finite update and its consumption, rather than acquisition of those arithmetic operations by a language model.
B’s useful multiplication call receives B’s negative terminal advantage. The total group can nevertheless favor multiplication because it combines that contribution with A’s success and the failed addition choices. Terminal weighting therefore supplies learning pressure to intermediate decisions without separately assessing the usefulness of each decision. ToRL v1, §§2–3 trains tool-interleaved math solutions with final-answer reward, retaining interpreter output as context while masking it from loss; its tested execution-error penalty was omitted from the main recipe. Search-R1 v5, §3 and Appendix B.2 likewise retains retrieved observations while optimizing generated output from terminal feedback. Its main implementation uses PPO, with GRPO as an alternative. These sources support the separation of observation, generated decision and outcome signal. The two-logit objective above is an explicit receiving construction.
Aggregation changes that construction. Including the tool return as a third item in this example’s averaging denominator would multiply each gradient by 2/3, producing probability 0.507216 rather than 0.510824; the displayed draws would still yield 31. Alternatively, form a product of the two action ratios and clip that whole-episode ratio. With the same four rewards, its first gradient is twice the displayed gradient, giving probability 0.521637 at rate 0.3. Subsequent whole-episode clipping also differs from clipping each decision. Choose and document the objective the trainer implements. In a language model, token lengths, generated-token normalization and observation masks must agree with that objective. Lengthening a return leaves this finite calculation unchanged only because its representation consumes the same integer observation and still has two generated decisions.
For further training, evaluate current-to-old ratios, clipping and any regularization as implemented by the selected trainer. Resample under a new rollout policy when that recipe requires it. This first-step derivative cannot be repeated as though every later ratio stayed 1. Retain a recoverable baseline, decide when learned parameters persist or reset, and assess changed and older useful behavior on separate cases as in section 26.4.