Engineering.Reference:26.4.2.3 - Execute the selected continuations and prepare their learning records
Use the same task (n*m)+k and exact calculator as in 26.4.1. For this run, the old policy selects multiplication with probability 0.9 and the unshifted answer observed+k with probability 0.5. Each choice is one generated position with a binary vocabulary. The first-position entropy is 0.325082973 nats; the answer-position entropy after the tool return is 0.693147181. With K=1 and V=2, delta_H is 0.184032104. The analytic policy supplies both full distributions; no sampled look-ahead or extra tool execution is needed to obtain them in this fixture.
Take M=4, N=2, alpha=tau=0.5 and beta=0.2. Two supplied draws, 0.2 and 0.95, select multiplication and addition on (17,19,5). Execute those calls, obtaining 323 and 36. Each saved history now needs only an answer, so an additional continuation needs zero further tool calls in this task. Consider proposals from the first lineage, that same lineage again, and the second lineage. Their scores are 0.536806421, 0.429445137 after the first acceptance, and 0.536806421. Accepting the first and third proposals spends the two reserved places and avoids assigning both to the repeated lineage. Assess this trial priority against fresh starts or a fixed split on the receiving tasks.
Finish each original and its one child using answer draws 0.1 and 0.9. The executed finite environment supplies these records:
| Saved history | Continuation answer | Terminal reward from the task checker | Old probabilities of the two selected decisions |
|---|---|---|---|
| Multiplication returned 323 | 328 | 1 | (0.9, 0.5) |
| The same multiplication prefix | 329 | 0 | (0.9, 0.5) |
| Addition returned 36 | 41 | 0 | (0.1, 0.5) |
| The same addition prefix | 42 | 0 | (0.1, 0.5) |
Two calculator executions supplied four completed trajectories. Descendants of a saved prefix share the same draw and tool result, so their training records are dependent. For each branch, obtain its later observations and qualified feedback. When those are unavailable, the sampler has produced an allocation or a prefix, not a completed learning example. Return that missing execution or feedback before assigning the branch a reward.
Choose a mean over the two generated decisions of each of these four trajectories. Keep the actual return in the answer decision’s history and exclude it from both the output loss and this denominator. Use the population-standard-deviation convention of 26.4.1: rewards (1,0,0,0) give advantages (sqrt(3),-1/sqrt(3),-1/sqrt(3),-1/sqrt(3)). The code uses zero advantages when reward dispersion is at most 1e-12, including equal rewards. This convention differs from a sample-standard-deviation implementation, which gives (1.5,-0.5,-0.5,-0.5). Choose one with the learning rate and aggregation that consume it. Reweighting a shared prefix or retaining it only once also changes this construction; averaging descendant advantages is not generally equivalent under signed clipping and unequal lengths.