Engineering.Reference:26.4.2.4 - Select modulation and the backward operation separately
A later training checkpoint may differ from the sampling policy. For the same records, take current multiplication probability 0.9 and current unshifted-answer probability 0.65. Keep the old probabilities and reward advantages fixed. For each generated position let r be its current selected-action probability divided by its old selected-action probability. The ordinary contribution to the maximized objective is min(r*A, clip(r,1-epsilon_low,1+epsilon_high)*A). Its sign matters: moving a negative advantage outside the minimum changes the operation.
An optional AEPO-inspired intervention forms A_tilde = A*(1 + a*z_H), where z_H standardizes the current token-distribution entropy. Choose the population for standardization. Here it consists of all eight generated positions, with population standard deviation, a=0.2 and zero standardized values when the deviation is at most 1e-12. The current call and answer entropies are 0.325082973 and 0.647446639 nats, so their standardized values are −1 and +1. Their multipliers are 0.8 and 1.2. Calculate these values at the checkpoint and hold them fixed during differentiation, as with the advantages. Obtaining entropy for this operation is distinct from adding an entropy regularization term; an implementation must request the required probability information even when the regularization coefficient is zero.
The multiplier magnifies both positive and negative advantages at higher-entropy positions. A zero advantage stays zero. It can even reverse the reward signal: one zero-entropy position among one hundred positions with entropy log(2) has population-standardized value −10, giving multiplier −1 at a=0.2. This receiving construction stops the modulation branch whenever a multiplier is nonpositive. The unmodulated update remains available. A positive exponential multiplier, a bound or another normalization would be a separately chosen modification. This application chooses both the stop and the neutral treatment of tiny dispersion.
The second intervention changes how an upper-clipped positive term contributes to the update. Define stop_gradient(r) to return r’s current numerical value while holding that denominator constant during differentiation. Replace the constant upper bound by (1+epsilon_high)*r/stop_gradient(r). The complete term is:
J_t = min(r*A_tilde,
clip(r, 1-epsilon_low,
(1+epsilon_high)*r/stop_gradient(r))*A_tilde)
Its forward value equals ordinary clipping at the checkpoint. Its backward operation differs. Away from ties, the coefficient multiplying A_tilde * grad(log current_probability) is:
| Condition | Ordinary constant-bound clipping | Stopped-denominator upper bound |
|---|---|---|
| A_tilde > 0 and r > 1+epsilon_high | 0 | 1+epsilon_high |
| A_tilde < 0 and r < 1−epsilon_low | 0 | 0 |
| Otherwise | r | r |
To check the selected backward rule by finite differences, perturb the numerator while holding the stopped denominator and entropy weights fixed at the original checkpoint. Recomputing the stopped value at each perturbed point instead checks the ordinary forward function. The backward rule applies to every positive upper-clipped term, regardless of entropy. Entropy modulation can be enabled or disabled independently. This application uses neither an additional dual-clip rule nor a reference-policy penalty.
Keep the two clipping widths separate. For adjusted advantage −1.2, ratio 0.75, epsilon_low=0.2 and epsilon_high=0.28, the selected term is −0.96 and its gradient is zero. The lower test uses 0.8. Substituting the high width would test against 0.72 and incorrectly allow an update. This example follows the lower-width condition in AEPO’s appendix and the displayed objective. For a positive advantage above the upper bound, the ordinary rule has a plateau while the selected backward alternative continues to reinforce it. Preserving that distinction does not establish which intervention improves the receiving task.