Obtain labels for a rule that will be used on further cases

The next action can be another query rather than the final subject action. Use this branch when examples are available without their labels and obtaining a label can improve a rule used on further cases. The result is a policy: which label to request now, what to do with each possible answer, when to request another, and what rule to use when questioning ends. Use an adequate existing rule without further labeling when it already serves the receiving work.

Construct the query and its consequence together. First name the cases on which the rule will be used, the response it must give and the loss of a wrong response. A query that reveals much about a rare case can be less useful than a less uncertain query that changes many consequential predictions. Learning a rule and estimating the accuracy of a fixed rule are also different purposes; the same selected labels need not serve both.

Recover what is observable before asking. In a pool, the unlabeled cases are already available; in a stream, rejecting a case may lose the opportunity to label it. A synthesized input may not be interpretable or labelable by the person answering. Specify the available answer, its reliability and delay, and the full cost of requesting it, including preparation and waiting. If asking changes the object, include that transition as in the diagnostic connection above.

CMP.7 supplies an effective procedure for obtaining and updating the rule from examples. MMP.16 then compares requests through what that procedure and the receiving use would obtain. For each affordable candidate query, enumerate or otherwise model its possible answers, update the rule under each answer, and calculate the resulting loss on the receiving cases. Average over the answers that have not yet been observed; do not use the future true label as if it were available when selecting the query. Include the query’s cost in the same comparison as acting now.

Choose the organization that is feasible:

• One-step adaptive choice: compare the next query assuming the returned rule is then used. Repeat after each actual answer if further queries remain possible. This uses feedback but can miss a query whose value lies in its continuation.
• Conditional continuation: use MMP.8.SD to compare the present query together with later choices contingent on its answers. Retain the remaining requests, resources, timing and other conditions that change those choices, not just the current probabilities.
• Fixed batch: choose a set before receiving its answers. This can avoid waiting between requests, but pays for every member and cannot skip one made unnecessary by an earlier answer. Evaluate the set jointly; the highest individual scores can select redundant cases.

The full comparison may itself be too costly. Uncertainty, disagreement among plausible rules, expected change in a rule and coverage of the receiving cases are possible cheaper indicators. They estimate different things. Use one because its relation to the present use and cost is adequate, not because “most uncertain” defines “most useful.” Retain a way to investigate plausible regimes omitted by a confident but inadequate model. Preparing a sophisticated policy is worthwhile only when its improvement can repay that preparation as well as the queries.

A finite construction. A service will classify 100 documents: 95 of observable type X and 5 of type Y. Each wrong classification takes one minute to correct. Three candidate rules assign the following labels, with supplied initial probabilities:

These are stipulated teaching premises, not estimates or a claim about actual documents. Exactly one rule is assumed true, every document of a type has that type’s label, and the available specialist answers without error. Requesting X costs two minutes; requesting Y costs three. These are full costs. At most these two requests are available, and an answer can arrive before the next request and before classification. Acquisition does not alter the labels or subsequent correction costs.

Construct the learner by retaining the rules consistent with each answer and renormalizing their probabilities. For a type, predict the label with smaller conditional expected correction loss; choose label 0 if the losses tie. Applying the returned rule to the documents is distinct from computing these updates. If an answer leaves no consistent rule, this construction has failed its premises and must return to the model or answer account.

With no request, predict X as 0 and Y as 1. The expected correction time is

95 × 0.10 + 5 × 0.45 = 11.75 minutes.

Work backwards from the final classification choice. After each possible first answer, compare stopping with obtaining the remaining label. This constructs the value of a first request from its obtainable continuation rather than from its uncertainty alone.

Thus requesting X first costs 2 + 0.9 × 2.5 = 4.25 expected minutes in total. Requesting Y first with its conditional continuation costs 3 + 0.55 × 2 = 4.10. Select Y first under these premises and costs.

This choice is not obtainable by treating every first request as the last. Y alone leaves 9.5 expected minutes of correction, so its total of 12.5 is worse than the original 11.75. A one-step comparison chooses X instead. A fixed batch of X and Y costs 5 and settles both labels, but purchases X even when Y=0 already identifies h1. The conditional policy saves that unnecessary request. These numbers compare the stated acquisition and correction costs; if constructing the policy adds material work, add that burden before adopting it over the simpler choice.

Use the actual answer. If the specialist returns Y=1, retain the updated probabilities 9/11 and 2/11, ask X, and apply the corresponding rule. If Y=0, apply h1 without another query. Record and use the answer actually obtained; a predicted branch is not evidence that its label has been received. If the answer arrives too late for another query, the conditional policy is unavailable. With only one request allowed and the other premises unchanged, choose X, whose total remains 4.25. A feasible alternative that returns both batch answers in time can still be compared at its full cost.

Recalculate what changed. Suppose the full price of Y rises from 3 to 3.2 minutes. The response probabilities do not change. The Y-first policy now costs 3.2 + 0.55 × 2 = 4.30, while X-first still costs 4.25, so choose X first. Y remains the more uncertain answer, but that fact no longer selects the better policy. After X=0, stop with Y unresolved: its remaining expected correction time is 2.5, less than the cost of settling it. This is a sufficient economic choice for this use, not knowledge of every label.

Changed proportions of X and Y, correction losses, time available or permissible predictions likewise reopen the affected comparison. Repeating an already answered type adds no information under the noiseless, constant-label premises. Noisy or dependent answers require their actual likelihood and a new update; a second answer cannot be treated as independent merely because it is a second request. A later observation of X=1 and Y=0 contradicts every rule in the example. Check the labels and the type definition, then revise the family or the response model through MMP.14 rather than continuing to report a zero model risk.

Keep stopping and further-use evidence distinct. A stopping decision can mean that the present rule is adequate, that additional expected benefit does not repay its cost, or that resources are exhausted while a required result remains unresolved. Return which of these holds. A narrow cost comparison does not establish a broader required accuracy level.

The example’s expected losses are conditional on its supplied class, probabilities and constant labels. They are not measured accuracy on actual future documents. For an empirical accuracy claim, use assessment cases and a justified uncertainty account suited to the receiving population. Keep them separate from cases used to fit, select or repeatedly tune the rule. The average error on adaptively requested labels is generally not the receiving population’s average: requests were chosen for another purpose. MMP.13 preserves the selection and stopping law in an inference. If assessment is itself adaptive, use a method qualified for that design; an ordinary fixed-sample interval or a naive reweighting does not supply that qualification. MMP.17 supplies a related adaptive construction when queries obtain responses of a costly source model, while keeping agreement with that model distinct from agreement with the world.

Settles’s Active Learning Literature Survey, §§2–4 and 6, develops query settings, alternative selection criteria and their practical limitations. Huan, Jagalur and Marzouk, §5, develops the distinction between fixed designs and policies that use intermediate observations. Farquhar, Gal and Rainforth shows why correcting an adaptively selected risk estimate and improving the learned predictor are different questions. The construction here uses a small explicit family; it does not supply those papers’ specialized estimators or guarantee improvement for arbitrary learned models.