Engineering.Reference:26.4.2.5 - Run the finite construction and use the changed policy
The code executes the two calculator calls and four answer continuations, checks the results and constructs the reward-relative update. Its two bits are the selected generated positions; the actual return remains in their record and contributes no output term. It keeps the population convention, mean over eight terms, detached modulation and ascent rate 0.1 stated above. The repeated proposal is at the same saved prefix; its changed counter, rather than new probability evidence, changes admission. Terminal children need no further counter decisions in this fixture.
from math import exp, log, sqrt
def sigmoid(z):
return 1 / (1 + exp(-z))
def entropy(probabilities):
assert all(p >= 0 for p in probabilities)
assert abs(sum(probabilities) - 1) < 1e-12
return -sum(p * log(p) for p in probabilities if p)
def standardized(values):
mean = sum(values) / len(values)
sd = sqrt(sum((x - mean)**2 for x in values) / len(values))
return [(x - mean) / sd for x in values] if sd > 1e-12 else [0.0] * len(values)
old = (0.9, 0.5) # Multiplication, then the unshifted answer.
request = (17, 19, 5)
baseline_H = entropy((old[0], 1 - old[0]))
after_H = entropy((old[1], 1 - old[1]))
delta_H = (after_H - baseline_H) / 2 # One position, vocabulary size two.
# Execute two starts. The exact calculator is retained external support.
roots = {}
training_calls = 0
for name, draw in (("A", 0.2), ("B", 0.95)):
multiply = int(draw < old[0])
n, m, k = request
observed = n * m if multiply else n + m
training_calls += 1
roots[name] = {"bit": multiply, "observed": observed, "forks": 0}
# Only parent-lineage counters are needed before these terminal continuations.
reserve = 2
allocation = []
for name in ("A", "A", "B"):
lineage = roots[name]
score = (0.5 + 0.2 * delta_H) * max(0, 1 - 0.2 * lineage["forks"])
accept = reserve > 0 and score > 0.5
allocation.append((name, round(score, 9), accept))
if accept:
lineage["forks"] += 1
reserve -= 1
assert [r[2] for r in allocation] == [True, False, True]
assert reserve == 0 and training_calls == 2
# Each saved prefix now has its original and one new continuation.
# Only answers remain in this fixture, so forks require no further tool call.
records = []
for name, prefix in roots.items():
assert prefix["forks"] == 1
for draw in (0.1, 0.9):
answer_bit = int(draw < old[1])
answer = prefix["observed"] + request[2] + (1 - answer_bit)
records.append({
"histories": (request, (request, ("call", prefix["bit"]),
("return", prefix["observed"]))),
"bits": (prefix["bit"], answer_bit),
"old_probabilities": (old[0] if prefix["bit"] else 1 - old[0],
old[1] if answer_bit else 1 - old[1]),
"answer": answer,
"reward": int(answer == request[0] * request[1] + request[2])})
assert [r["reward"] for r in records] == [1, 0, 0, 0]
# A later training checkpoint, evaluated on the same old-policy records.
current = (0.9, 0.65)
logits = [log(p / (1 - p)) for p in current]
advantages = standardized([r["reward"] for r in records])
position_H = [entropy((p, 1 - p)) for p in current]
all_H = position_H * len(records) # Eight generated positions; no return token.
z_H = standardized(all_H)
def update(modulate=False, follow_upper=False):
count = 2 * len(records)
gradient = [0.0, 0.0]
objective = 0.0
for i, record in enumerate(records):
for j, bit in enumerate(record["bits"]):
multiplier = 1 + 0.2 * z_H[2*i + j] if modulate else 1.0
if multiplier <= 0:
raise ValueError("Linear modulation would reverse or erase the signal")
advantage = advantages[i] * multiplier
p = current[j]
probability = p if bit else 1 - p
ratio = probability / record["old_probabilities"][j]
objective += min(ratio * advantage,
max(0.8, min(1.2, ratio)) * advantage) / count
if advantage > 0 and ratio > 1.2:
coefficient = 1.2 if follow_upper else 0.0
elif advantage < 0 and ratio < 0.8:
coefficient = 0.0
else:
coefficient = ratio
gradient[j] += coefficient * advantage * (bit - p) / count
changed = [sigmoid(z + 0.1*g) for z, g in zip(logits, gradient)]
return objective, gradient, changed
def act(probabilities, request, draws):
n, m, k = request
observed = n * m if draws[0] < probabilities[0] else n + m
return observed + k + (0 if draws[1] < probabilities[1] else 1)
print("allocation", allocation)
for modulate, follow_upper in ((False, False), (True, False), (False, True), (True, True)):
objective, gradient, changed = update(modulate, follow_upper)
print(modulate, follow_upper, round(objective, 9),
[round(x, 9) for x in gradient], [round(x, 9) for x in changed])
assert act(update()[2], (12, 13, 5), (0.5, 0.65)) == 162
assert act(update(True, True)[2], (12, 13, 5), (0.5, 0.65)) == 161
Each row starts from the same current checkpoint and uses the same old-policy records:
| Entropy modulation | Upper-bound backward operation | Mean forward value | New (multiplication, unshifted-answer) probabilities |
|---|---|---|---|
| Off | Ordinary clipping | 0.050518149 | (0.901291559, 0.649252595) |
| On | Ordinary clipping | 0.060621778 | (0.901034441, 0.649103026) |
| Off | Stopped denominator | 0.050518149 | (0.901291559, 0.651320527) |
| On | Stopped denominator | 0.060621778 | (0.901034441, 0.651584354) |
For the last row, the mean update direction is (0.115470054,0.069715045). With ordinary clipping and no modulation it is (0.144337567,-0.032836797). The successful unshifted answer is upper-clipped, while a failed unshifted answer contributes a negative term; this explains the ordinary answer probability’s decrease in this batch. The alternative backward operation restores the successful term’s positive contribution. The table also shows why equal forward values cannot identify the implemented backward operation.
Both changed policies use the calculator. With the new request (12,13,5) and stipulated draws (0.5,0.65), the ordinary update returns 162, while the last row returns 161. Reset the request history and retain the chosen parameters and tool. This exhibits consumption of the finite update; the chosen draws and one conditional batch do not estimate effectiveness. The independent arithmetic operations, interface meanings and task checker are supplied capabilities. A language-model implementation must accumulate the selected generated-token contributions through its actual shared parameters, execute its optimizer and retain the support on which its later behavior depends.
Change the probability-access condition before generalizing the example. The binary full distributions above make its entropy calculations possible. If a four-outcome API instead supplies only selected probabilities, use the ranges consistent with those probabilities. Disjoint ranges settle an entropy ranking. Where both orders remain possible, the same data can still support recorded action ratios. Preserve the available outcome update or fixed allocation, and obtain the missing information only if the intended intervention justifies its cost. Change the branch history separately: the same measured delta_H can admit an unbranched lineage and reject another with l=1.