CMP.7:5.1 - Construct a batch selector and expose an unresolved continuation

On nonnegative integer inputs, consider the two rules f(x)=x and g(x)=x mod 2. The supplied examples are (0,0) and (1,1). Both rules have zero squared loss on those examples.

A learner can evaluate both rules and return the minimizing set {f,g}. This is an effective obtained result with retained ambiguity. It can also return one rule under a declared preference, but the zero training loss alone gives no reason to prefer f over g at input 2. Their predictions there are 2 and 0.

Suppose the receiving task needs a prediction at 2 and an additional observed response is available for a cost that could be justified. If that response is 2, the same selection procedure prefers f; if it is 0, it prefers g. If the observation is not worth obtaining, use a selected rule under the remaining assumption or retain both possible responses for the downstream decision. Better minimization of the original two-example loss cannot distinguish them.

Changed response: suppose the response at 2 is 1. Neither rule fits all three observations. Revisit the two-rule family or the assumption of deterministic noiseless responses. Repeatedly fitting the original family more accurately cannot create the absent continuation.