KCAE.EVAL:4.2 - Build episodes that can defeat the design

Use actual receiving episodes where access is permitted, or independently construct realistic cases from work conditions. Retain the original request, relevant source basis, available case facts and expected kind of useful result. Separate cases used to design queries, tune thresholds or write features from cases used to assess them. Prevent near-duplicate leakage across that boundary.

Include cases with unknown source names, cross-language wording, useful non-pattern prose, distant prerequisites, misleading vocabulary, an important exception, missing facts and no useful corpus contribution. Include naturally answerless cases from the receiving work, not only deliberately irrelevant collections. Add source changes, stale indexes, revoked access and partial availability when the intended installation must handle them. Select proportions from the expected workload or report separate strata; a hand-balanced set is not automatically representative.

Gold material may be incomplete. Pool candidates from several serious routes and obtain source-based judgements from appropriate readers; sample beyond the pool where practical. Record uncertainty about unjudged material. Do not label every unpooled passage irrelevant. When several answers can serve the work, judge sufficiency and conditions rather than exact string agreement with one preferred answer.

Synthetic questions generated from an index can be useful development probes. Keep that origin visible and do not use them as the only evidence that the same index finds unfamiliar distinctions. For an English-primary assessor serving another language, test that language and important negation/condition forms rather than assuming a good English average transfers.

For a broad synthesis, vary the underlying evidence independently of its presentation. Duplicating an old edition or adding an overlapping summary must not create another population member or inflate a count. A new current counterexample or changed qualifying note must be able to alter the claim it defeats. Inspect whether a rare consequential position survives the map, reduction and delivery, and whether unknown membership remains unknown. Compare the final claims with original-source coding on a manageable dossier; merely counting processed summaries cannot test this contribution. KCAE.SEARCH:5.2 provides a constructed example, not an empirical success rate.