Engineering.Reference:26.4.1.1 - Keep the generated choices and their available histories
The task is to return (n*m)+k. The supplied calculator evaluates multiplication or addition exactly. A finite policy has two trainable scalar logits, initially zero. The first chooses n*m with probability sigmoid(z1) and n+m otherwise. After receiving the calculator’s return o, the second chooses the answer o+k with probability sigmoid(z2) and o+k+1 otherwise. The second logit is shared across these histories, but its answer uses the observation returned for that history. The input supplies n, m, k and the calculator’s meaning. The arithmetic operations and the available choices are provided capabilities; only their choice probabilities are being learned.
The worked example stipulates the following four recorded episodes, sampled in this deterministic fixture with the same unchanged old policy. Each generated choice had probability 0.5. The calculation uses these four episodes. Another sample need not include all four combinations.
| Episode | Generated calculator call | Observed return | Generated answer | Independently checked terminal reward |
|---|---|---|---|---|
| A | 17*19 | 323 | 328 | 1 |
| B | 17*19 | 323 | 329 | 0 |
| C | 17+19 | 36 | 41 | 0 |
| D | 17+19 | 36 | 42 | 0 |
For B, the first learning record contains the request, tool meaning, selected multiplication call and its old probability. The second contains that same request, the call the policy selected, its observed return of 323, the selected answer 329 and its old probability. The result checker supplies reward 0 outside those decision inputs. It does not put the correct answer into the policy’s earlier history. Preserve the producing configuration and the observations needed to interpret each record.
Select the two generated decisions as policy-output positions. The calculator return stays in the second decision’s input and has no policy-output target of its own. For a language model, the same distinction concerns generated tokens and externally supplied tokens; the selected trainer must implement both the mask and the intended loss normalization. A tag that the model can forge does not authenticate a tool result. Obtain observations through the execution interface and preserve their provenance. A missing or altered return requires repairing the experience or defining the resulting different input condition.