Jump to text
On this page
Copy and source
Published source confirmed at last check

Source changed 2026-11-22 15:54:17 UTC · snapshot created 2026-11-22 15:55:11 UTC · last check 2026-11-22 16:10:10 UTC

Engineering.Reference:26.4.1.1 - Keep the generated choices and their available histories

The task is to return (n*m)+k. The supplied calculator evaluates multiplication or addition exactly. A finite policy has two trainable scalar logits, initially zero. The first chooses n*m with probability sigmoid(z1) and n+m otherwise. After receiving the calculator’s return o, the second chooses the answer o+k with probability sigmoid(z2) and o+k+1 otherwise. The second logit is shared across these histories, but its answer uses the observation returned for that history. The input supplies n, m, k and the calculator’s meaning. The arithmetic operations and the available choices are provided capabilities; only their choice probabilities are being learned.

The worked example stipulates the following four recorded episodes, sampled in this deterministic fixture with the same unchanged old policy. Each generated choice had probability 0.5. The calculation uses these four episodes. Another sample need not include all four combinations.

EpisodeGenerated calculator callObserved returnGenerated answerIndependently checked terminal reward
A17*193233281
B17*193233290
C17+1936410
D17+1936420

For B, the first learning record contains the request, tool meaning, selected multiplication call and its old probability. The second contains that same request, the call the policy selected, its observed return of 323, the selected answer 329 and its old probability. The result checker supplies reward 0 outside those decision inputs. It does not put the correct answer into the policy’s earlier history. Preserve the producing configuration and the observations needed to interpret each record.

Select the two generated decisions as policy-output positions. The calculator return stays in the second decision’s input and has no policy-output target of its own. For a language model, the same distinction concerns generated tokens and externally supplied tokens; the selected trainer must implement both the mask and the intended loss normalization. A tag that the model can forge does not authenticate a tool result. Obtain observations through the execution interface and preserve their provenance. A missing or altered return requires repairing the experience or defining the resulting different input condition.