Jump to text
On this page
Copy and source
Published source confirmed at last check

Source changed 2026-11-22 14:27:11 UTC · snapshot created 2026-11-22 14:28:00 UTC · last check 2026-11-22 15:05:20 UTC

Engineering.Reference:26.4.2.2 - Allocate attempts and reserve the work needed to finish them

Separate a count of complete training trajectories from resource limits such as generated tokens, tool calls and elapsed time. Set aside resources to finish and qualify the original attempts before admitting branches. For example, two extra trajectory places, six extra tool calls and an upper bound of three calls per continuation admit at most two new continuations. Three calls cover the full individual allowance for one. Two can fit when their joint remaining demand is at most three and two trajectory places remain. Include retries, probability extraction and any pilot in the resource account. Shared prefixes may save work, but their existence does not bound the cost of their continuations. SYSE.51’s budgeting principles help retain completion resources; the training sampler described here is a further selected construction.

One ARPO-inspired receiving rule starts N trajectories and reserves M−N of M complete trajectories for branches. Measure K initial generated positions and K positions after actual tool feedback. For the conference paper’s literal normalization, define delta_H = sum_t(H_after,t - H_initial,t)/V, where V is vocabulary size. This is a scaled change, not the average over K. Choose alpha, beta and a threshold tau; branch when alpha + beta*delta_H > tau and both count and resource capacity remain. This strict comparison gives a deterministic decision. For a randomized decision, compare a random draw with a branch probability in [0,1]. The two rules can make different decisions even when delta_H is zero.

To limit repeated allocation to a lineage, this application adds AEPO’s proposed linear penalty. Let l count consecutive accepted branches with positive delta_H along that lineage. Start l at zero, use score = (alpha + beta*delta_H)*max(0,1-0.2*l), and require score > tau. On accepting a positive-change branch, increment l for both the continued parent and its new child. Reset it on a later measured transition with nonpositive delta_H. This application chooses those counter and reset conventions. Keep counters per lineage; a fresh start uses a new counter. Finish existing attempts when capacity is exhausted; unused trajectory places may return to fresh starts only when those starts and their completion fit the remaining resources.

AEPO also proposes choosing the split with a pilot. With B available training trajectories after charging the pilot, its paper rule is m_raw = B*sigmoid(beta*(H_root - H_tool_average)); higher initial entropy favors fresh starts, and higher post-tool entropy favors branches. Specify comparable windows and the averaging rule. This example chooses m = min(B,max(1,ceil(m_raw))) and reserves B−m branch places. For B=16, beta=0.2, H_root=1 and H_tool_average=2 nats, m_raw is 7.202656, giving eight starts and eight branch places. The paper proposes the sigmoid as an allocation heuristic. Its separate linear information-gain proxy does not derive that rule.

The pilot must be affordable before it can inform that choice. Suppose the whole budget permits 48 calls, the completed pilot used three, and each retained training trajectory may require up to three further calls. A conservative count bound is now B=15, so the same rule gives seven starts and eight branch places. Reserving the individual maximum for sixteen trajectories plus that pilot takes 51 calls. With 45 remaining after the pilot, fifteen are covered on those bounds. To exclude sixteen, establish a binding reservation rule or a conflict with their actual joint requirements. For a tighter allocation, use qualified estimates of the resources each remaining continuation needs. When the pilot or probability extraction cannot fit, the fixed split remains an available baseline. The finite run below uses that fixed split and needs no pilot.