A.19.ECS - Evaluation CharacteristicSpace Construction: Define What Counts as Better
Type: Method pattern Status: Stable Normativity: Normative
Use this pattern when. Use this pattern when an object version is to be improved or judged, but the evaluation that says what “better” means is not yet available, not yet explicit, or not yet adequate for the object.
A.19.ECS:1 - Problem frame
Use A.19.ECS when an object version is to be improved or judged, but the evaluation that says what “better” means is not yet available, not yet explicit, or not yet adequate for the object.
A.19 says how a CharacteristicSpace is structured: declared characteristics, declared scales, slots, value sets, declared coordinate groups, and no hidden normalization or aggregation. A.19.ECS says how to make such a CharacteristicSpace for the evaluated object, so that an evaluation can later evaluate that object and E.23 can run an improvement loop without inventing values.
The ordinary output is an evaluation characteristic-space specification: a grouped set of characteristics, scales, value meanings, evidence-basis rules, missingness rules, result-row shape, calibration points, coordinate-specific evidence payloads, protected trade-offs, status meanings, and stop or reopen conditions for one evaluated object kind and use scope.
Not this pattern when. If a suitable evaluation already exists, cite it and use E.22 for question framing or E.23 for repeated improvement. Use A.17, A.18, and C.16 when the live problem is one characteristic, one scale, or measurement admissibility and scale lawfulness. Use C.16.P first when candidate coordinate wording still hides whether the use under repair is a characteristic, scale, coordinate, score, metric label, quality-term repair, or subject-pattern relation. Use A.19 when the live problem is the structure of CharacteristicSpace itself. Use C.25 when the evaluated EntityOfConcern is a composite engineering quality family that already fits Q-Bundle form. Use F.18 when the live problem is durable naming. Use E.21, E.9.DA, or E.2.DA when the evaluated EntityOfConcern is respectively one FPF pattern version, one DRR, or one FPF-level Pillar-adequacy evaluated EntityOfConcern.
First useful move. State the sentence: “good as what kind of object, for which use, against which contrast cases?” Then name the evaluated object kind, the use scope, and at least three contrast cases: one admissible evaluated object, one below-floor evaluated object, and one outside-declared-object-kind boundary case that should return to evaluation selection before the evaluation is opened or receive an explicit object-kind-fit defect/value when that evaluation has already been invoked.
Existing-evaluation boundary. If the answer is “use this existing evaluation” and the evaluated object kind, use scope, floor, protected trade-offs, and stop meanings are already recoverable, do not construct a new CharacteristicSpace.
What goes wrong if missed. A team says “improve this” and then chooses convenient scores. A scale set appears from nowhere. Chairs, coal plants, nuclear plants, and FPF patterns all get compared on coordinates that do not distinguish the evaluated object kind. One visible value improves while the intended use gets worse. A review can say “better” but cannot say which object property changed, what trade-off was protected, or why improvement may stop.
What this buys. A.19.ECS gives improvement work a way to create the missing evaluation before the loop starts. It keeps E.23 universal and simple: E.23 changes the object and asks an evaluation to re-evaluate it; A.19.ECS helps build that evaluation when none is yet adequate.
Primary EntityOfConcern in plain terms. The primary EntityOfConcern is the construction of one evaluation CharacteristicSpace for one evaluated object kind and declared use.
Primary working reader. The first reader is the engineer, analyst, pattern author, evaluator, steward, or method designer who must define what counts as improvement for an evaluated object before running an improvement loop.
A.19.ECS:2 - Problem
FPF already has named patterns for single characteristics, scales, coordinate values, Q-Bundles, and repeated improvement. The gap is the construction of a useful grouped scale set for an evaluated object kind.
Recurring failures:
- Scale set from air. An evaluation lists coordinates because they are familiar, not because they discriminate the evaluated object kind or use.
- Wrong-kind comparison. Objects outside the declared kind are scored as if they were weak objects under improvement, or are silently skipped, instead of being returned to evaluation selection before opening or handled by an explicit object-kind-fit defect/value after opening.
- One-score collapse. Several independent characteristics are averaged into one score, hiding object-kind-fit defects and trade-offs.
- Unstated polarity. Readers cannot tell which direction is preferred or when a value has no preferred direction.
- No floor or exceptional meaning. Values are recorded, but nobody can say what is viable, exceptional, or still inadmissible for the declared use.
- No evidence or missingness rule. A coordinate value is asserted without saying what observation, content locus, test, example, source, or judgment can justify it, or what absence means.
- No protected trade-off. The evaluation encourages improvement on visible coordinates while damaging safety, usability, affordability, source preservation, entry cost, neighbour fit, or another value that should constrain the change.
- No stop or reopen condition. Improvement continues forever or stops after a convenient checklist closure, not because the evaluation says the evaluated object has reached the declared aim.
- Specification underdeclaration. A new evaluation is mentioned in prose, table, rule, or local rubric, but its declared specification does not make evaluated object kind, coordinate set, value meanings, status meanings, relations, and non-use boundaries recoverable.
- Result-form underdeclaration. The evaluation has coordinates, but the returned result can be a prose impression, a two-column value table, or a checklist count without evidence basis, adjacent-value rationale, calibration discipline, or coordinate-specific payload.
- Evidence-basis leakage. Evidence needed to justify the evaluation result, corpus projection, currentness, retrieval, or parity is written as if it were the evaluated object’s own method or user action.
A.19.ECS:3 - Forces
| Force | Tension |
|---|---|
| evaluated-object-kind discrimination vs broad reuse | The evaluation must fit the evaluated object kind, but it should reuse existing FPF characteristic and scale discipline where possible. |
| Small first version vs enough coordinates | A useful first evaluation can be compact, but it needs enough coordinates to block false improvement and wrong-kind comparison. |
| Measurement admissibility and scale lawfulness vs ordinal judgment | Some coordinates are measured through C.16; others are evidence-backed ordinal content values. The evaluation must say which is which. |
| Improvement direction vs trade-off protection | Preferred movement must be visible without turning every coordinate into an optimization command. |
| Contrast cases vs overfitting | Contrast cases are needed to test the scale set, but the evaluation must not become a list of examples only. |
| Reusable specification vs local use | A reusable evaluation must make the same evaluation characteristic-space elements recoverable across uses. A local project can use a smaller specification when the use is bounded and non-reusable. |
| Local stop vs open-ended improvement | A loop may stop for the declared use while the object and the scale set remain improvable under a new use, source, or comparison concern. |
A.19.ECS:4 - Solution
Construct an evaluation CharacteristicSpace by declaring the evaluated object kind, use scope, contrast cases, characteristic slots, scale bindings, value meanings, evidence-basis and missingness rules, result-row shape, calibration points, coordinate-specific evidence payloads, protected trade-offs, status meanings, and stop or reopen conditions.
EvaluationCharacteristicSpaceSpec := <EvaluatedObjectKindRef, ObjectVersionUnderImprovementRef?, DeclaredUseScope, WorkingReaderScope, QualificationWindow, DiscriminatingCaseSet, ObjectKindFitRule, CharacteristicSlotSet, ScaleBindingSet, PolarityAndPreferredMovement, FloorAndExceptionalMeaningSet, EvaluationEvidenceBasisRule, EvidenceAndMissingnessRule, ResultRowShape, AdjacentValueRationaleRule, CalibrationPointSet, CoordinateSpecificEvidencePayloadRule?, ProtectedTradeoffSet, DominanceOrComparisonRule?, StatusValueSet, StopOrReopenCondition, NeighborPatternExitSet, E22QuestionFrameUse?, E23StartCondition>
A.19.ECS:4.1 - Local names and kind settlement
| Local name | Use | Non-use boundary |
|---|---|---|
EvaluationCharacteristicSpaceSpec | Local specification for constructing one evaluation CharacteristicSpace. | Not a score sheet, review packet, work plan, gate, evidence record, or project approval. |
EvaluatedObjectKindRef | Exact kind of object the evaluation evaluates. | Not a vague artifact, file bundle, campaign, chat, or source collection. |
DeclaredUseScope | Use for which the evaluated object is being judged or improved. | Not all possible uses. |
DiscriminatingCaseSet | Positive, below-floor, and outside-declared-object-kind boundary cases used to test whether the characteristic space distinguishes the evaluated object kind and use. | Not a substitute for the coordinate set. |
ObjectKindFitRule | Rule for admissible evaluated object, below-floor evaluated object, and outside-declared-object-kind boundary case. | Not permission to omit declared coordinates after an evaluation has been invoked. |
CharacteristicSlotSet | The grouped slots, each binding one characteristic to one scale. | Not an arbitrary checklist and not hidden aggregation. |
ScaleBindingSet | The chosen scale and value meaning for each characteristic slot. | Not a metric dashboard unless a distance or measurement claim is explicitly declared by the neighbour. |
PolarityAndPreferredMovement | Direction of preferred movement for each coordinate, or a statement that the coordinate has no simple preferred direction. | Not permission to optimize one coordinate while damaging protected trade-offs. |
FloorAndExceptionalMeaningSet | Viable-for-use and exceptional-for-use value meanings for declared coordinates. | Not a maturity ladder and not proof that future improvement is impossible. |
EvaluationEvidenceBasisRule | The checked evidence loci required for the result: object version, corpus/projection loci when corpus-facing, source-currentness loci when currentness is valued, comparator loci when parity is valued, worked-case loci when case coverage is valued, and any missing or unchecked basis that limits the conclusion. | An unchecked premise leaves its dependent value unestablished; it supplies neither a low property value nor a positive evaluation. Do not infer values from reputation, review state, or absence of visible defects. |
EvidenceAndMissingnessRule | What justifies a value and how missing, censored, unknown, object-kind-fit, or boundary-return cases are handled. | Not project evidence, assurance, or gate proof by itself. |
ResultRowShape | Required result row fields for the evaluation, including coordinate, value, and a short rationale; some evaluations may add evidence-locus or payload fields. | Not a free-form review paragraph and not a two-column coordinate/value table. |
AdjacentValueRationaleRule | Rule that each result rationale says why the lower adjacent value would understate the evidence and why the higher adjacent value would overstate it, or for the top value what would lower or reopen the claim. | Not verbosity for its own sake. |
CalibrationPointSet | Reusable 3/4/5 or equivalent adjacent-value calibration points for common evaluator disagreements. | Not a second score system and not a shortcut around the declared scale. |
CoordinateSpecificEvidencePayloadRule | Extra payload that a coordinate needs when a category label can fake discharge: comparator plus selected ingredient plus current locus, source plus adopted payload plus currentness window, projection locus plus retrieval cue, or another payload named by value. | Not administrative burden, not the evaluated object’s method, and not live evaluated-object text unless the evaluated object itself is an evaluation result or projection carrier. |
ProtectedTradeoffSet | Qualities or neighbour claims that must be checked when visible coordinates improve. | Not a hidden veto without a declared evaluation pattern or value meaning. |
PrecisionRepairKindRule | Rule for checking pre-repair and post-repair evaluated object kind, characteristic kind, relation or claim kind, current ontic slot, relation position, use relation, admissible use, and scope when coordinate or evaluation wording is repaired; when another pattern description contains the defining or constraining content, cite its subjectPatternLocator and exact ClaimGraph. | Not a lexical substitution table and not permission to change object kind or slot, relation position, use relation, or claim kind by cleaner wording. |
StatusValueSet | Local admissible-use result values for the evaluation. | Not release state, gate status, or evaluator praise. |
E23StartCondition | Minimum condition for using this evaluation inside E.23. | Not the improvement loop itself. |
These names are local to this pattern. They do not mint kernel U.* kinds, measurement templates, gate states, evidence kinds, or release states.
A.19.ECS:4.2 - Construction moves
Use these moves when constructing or repairing an evaluation. They are not a mandatory work sequence; each move is a required content question whose answer must be recoverable before the evaluation is used for improvement.
- Name the evaluated object kind and use. Say what object kind is being evaluated and for which declared use. If the evaluated object kind is not recoverable, stop before choosing coordinates.
- Build the discriminating cases. Include at least one evaluated object that should pass, one object of the same general family that should fail the floor, and one different object kind that should return to evaluation selection before opening or receive an explicit object-kind-fit defect/value if this evaluation has already been invoked.
- Choose candidate characteristics. Draw candidates from the object kind’s real failure modes, first-principles structure, user or operator harms, domain tradition, current
SoTA, existing evaluations, and FPF neighbouring patterns named by value. - Bind each slot. For each candidate, state the characteristic, chosen scale, value set, admissible domain, missingness semantics, and whether the value is a measurement claim or an ordinal content evaluation. Keep the property value, object-kind fit, and missing observation distinct. An observed absence of required support can justify a low value; an unperformed check cannot establish that absence. A local diagnostic code such as
0may represent missing basis only when explicitly declared as a code, without the arithmetic or comparison rights of a measured zero. Preserve already supported values while leaving dependent conclusions open. - Remove false coordinates. Drop coordinates that do not change admissible action, do not discriminate the evaluated object, duplicate another coordinate without a different repair action, or belong to another exact evaluation.
- Split compound coordinates. If a coordinate mixes two repair actions, two object kinds, or two incompatible scales, split it or assign one part to the neighboring pattern governing the claim that governs it.
- State preferred movement and trade-offs. For each declared coordinate, state the preferred direction or explain why no simple direction exists. Name the protected trade-offs that must be checked when the coordinate improves.
- Define result form, evidence basis, and calibration. State the required result row shape, evidence basis, adjacent-value rationale rule, calibration points for common disagreements, and any coordinate-specific payload needed for high or floor-reaching values.
- Define floor, exceptional, status, and stop. State the viable-for-use floor, exceptional-for-use meaning, status values, and local stop or reopen condition.
- Record subject assertions and their rule loci. When the coordinate depends on evidence, assurance, gate, work, decision, publication, naming, quality-bundle, measurement, OEE/NQD, or mathematical-lens content, name the exact subject, relation function, defining or constraining ClaimGraph, and subject assertion. A
subjectPatternLocatormay help find that ClaimGraph but asserts no governance relation; do not rewrite the dependency as routing or package-placement prose. - Start
E.23only after evaluation values exist. A repeated improvement loop can start only when the evaluated object version, evidence basis, result form, and evaluation are recoverable enough for re-evaluation.
A.19.ECS:4.3 - Evaluation specification minimum
A.19.ECS does not prescribe a publication or record form. It states which evaluation characteristic-space elements must be recoverable before an evaluation characteristic space is reusable for judgement or improvement. The selected publication or record form may be an FPF pattern, local engineering standard, rubric, table, review form, model card section, protocol note, or project rule, but that form is not governed here. The evaluation characteristic-space specification must make these items recoverable by value:
| Specification item | Required content |
|---|---|
Evaluation problem frame | Evaluated object kind, declared use, first useful move, existing-evaluation boundary, and what goes wrong if no evaluation exists. |
Non-use boundary | Boundaries to single-characteristic, measurement, Q-Bundle, naming, evidence, assurance, gate, work, decision, publication, and loop-method patterns. |
Local names and kind settlement | Local field names, use named by values, and non-use boundaries. |
Evaluation record shape | The local record or bundle shape used by the evaluation. |
Object-kind fit rule | Admissible evaluated object, below-floor evaluated object, and outside-declared-object-kind boundary handling before and after invocation. |
Evaluation evidence basis | Loci named by value that must be checked or named when a value depends on object version, corpus projection, source currentness, mature comparator, worked case, retrieval, or other external evidence. |
Result-row shape | Required result row fields, at minimum coordinate, value, and short rationale; any required evidence-locus or coordinate-specific payload fields are declared here. |
Coordinate set | Coordinate heads, properties of the evaluated object, evaluated-object properties and use conditions, scale/value meanings, evidence loci, and protected trade-offs. |
Calibration and payload rules | Adjacent-value calibration points and coordinate-specific payloads that prevent impressionistic 3/4/5 assignment or category-list discharge. |
Status and stop condition | Admissible-use statuses, local stop meanings, and reopen conditions. |
Worked slices | At least one passing evaluated object, one below-floor evaluated object, and one outside-declared-object-kind boundary case. |
Common anti-patterns | The false interpretations or values the evaluation must block. |
Neighbouring-pattern claim assignment | Neighbouring FPF patterns named by value and the claims being made that each pattern defines or constrains. |
This minimum is a content requirement, not a file-format requirement. For an FPF pattern publication form, E.8 still governs the authoring form. A.19.ECS only states what the evaluation must make recoverable so that E.22 can frame an improvement-oriented quality evaluation and E.23 can run a repeated improvement loop.
When construction or repair changes coordinate wording or evaluation wording, the evaluation characteristic-space specification records PrecisionRepairKindRule or an equivalent result-row requirement. The check compares the pre-repair and post-repair evaluated object kind, characteristic kind, relation or claim kind, current ontic slot, relation position, use relation, admissible use, and scope; when another pattern description contains the relevant definition or constraint, it cites that exact ClaimGraph and may add a non-semantic subject-pattern locator. A cleaner phrase that changes those items, treats a coordinate position as an object kind, or loses the value’s slot, relation position, use relation, or claim kind is a changed evaluation decision, not a wording repair.
A.19.ECS:4.4 - Discriminating-case test
An evaluation is not ready if it cannot distinguish these three outcomes:
- Admissible evaluated object. The object is of the evaluated object kind and can meet or exceed the floor under the declared use.
- Below-floor evaluated object. The object is of the evaluated object kind or a declared comparable family, but fails one or more floors.
- Outside-declared-object-kind boundary case. Before the evaluation is opened, the object should return to evaluation selection or construction rather than be treated as the evaluated object kind. If the evaluation has already been invoked for that object, the result is an explicit object-kind-fit defect/value or repair status, not omitted coordinates.
Example: for a nuclear-plant adequacy evaluation, a nuclear plant can vary along safety, output, maintenance, regulatory, thermal, waste-handling, grid, and resilience coordinates. A coal plant may be a power-generation alternative only when the declared use explicitly compares power-generation options across plant kinds. A chair or FPF pattern is outside the nuclear-plant evaluated-object kind: before opening the evaluation it returns to a suitable evaluation; after a forced invocation, the record shows an object-kind-fit defect/value rather than pretending the chair has weak nuclear-plant quality or silently skipping coordinates.
A.19.ECS:4.5 - Scale-set improvement
Improve an evaluation when its use misses a consequential defect, rejects an admissible object, recommends a harmful repair, fails on a new use, or demands more work than its decision value justifies. First distinguish a defect in the evaluation from a failure to perform it, unavailable inputs, or reliance beyond its scope. A better-written specification and more agreement among evaluators do not by themselves establish better decisions.
Use the following comparison to decide whether to adopt a change:
- Name the lost practical result. State which decision or next action the current evaluation gets wrong or cannot support, for which object and use. Preserve the current evaluation as the comparison basis and propose the smallest change that addresses that loss.
- Establish contrasting cases independently of the proposed evaluation. Use subject requirements, observed work results, or other applicable grounds to identify a consequential defect and a difficult but admissible case. Include a case in which an apparently helpful repair would damage a protected quality. The proposed evaluation’s own verdict cannot establish these cases’ correctness.
- Apply both evaluations to the same material. Keep object versions, task conditions and available evidence comparable. Inspect missed defects, false objections, the proposed action, damage from that action and the work needed to obtain and use the answer. Investigate disagreement through the conflicting grounds and conditions; neither majority agreement, stricter verdicts nor a larger finding count establishes practical improvement.
- Test adoption beyond the cases used for tuning. When the change was fitted to known cases, use a new case with an independent basis for the adoption decision. Keep its relevant answer out of tuning and disclose prior exposure or assistance. Replaying a known case remains useful development evidence; if no independent case is available, limit the conclusion to a trial in the examined scope.
- Choose at attainable cost. Use
C.11.DUAto compare the useful decision change with preparation, data collection, independent judgement, interpretation, repair, repeated evaluation, maintenance, transition and displaced useful work. Keep uncertain and non-commensurable costs visible. Adopt for the supported use, retain on a bounded trial, revise, reject, or keep the existing adequate evaluation. - Preserve scoped results. Name the changed evaluation and whether earlier results remain comparable, need an explicit bridge, or cannot support the new use. Re-evaluate only conclusions that depend on the changed rule or conditions. A new evaluation does not erase an earlier result within its supported scope.
This comparison checks the evaluation’s practical contribution. Use E.21 separately when the specification is an FPF pattern whose quality is in question; E.9.DA for the decision record selecting it; E.2.DA for FPF-level Pillar adequacy; F.18 for naming; and C.16, A.17, A.18, or A.19 for measurement, scale or characteristic-space admissibility. Their results answer those questions and can supply premises here. Use E.23 when repeated improvement of the evaluation is needed.
An independent subject basis and a bounded adoption comparison can settle the current choice without creating an endless sequence of numeric meta-evaluations. If a decisive premise remains unknown, retain the corresponding limit or trial disposition; another score does not supply that premise.
Worked comparison. A team proposes replacing a review criterion that counts source links with one that checks whether a required claim is actually supported. A known defective text has many links but omits the dependent claim; a difficult admissible text uses one sufficient source and a different valid explanation. The new criterion detects the omission and preserves the admissible explanation. A proposed repair that copies every source paragraph would make ordinary use harder, so the comparison also checks the repaired text. These are development results on known cases. Adoption still needs an independently grounded new case and an affordable way to obtain the supporting judgement. If the new criterion merely produces more objections or requires whole-corpus reading for every local use, revise it or retain the adequate earlier procedure in its supported scope.
A.19.ECS:5 - Archetypal Grounding
Show, FPF pattern quality. The evaluated object kind is one FPF pattern version. The existing evaluation is E.21, so A.19.ECS stays closed unless E.21 itself is being redesigned. E.23 may improve the pattern version under E.21.
Show, DRR adequacy. The evaluated object kind is one DRR version for a declared campaign-decision use. The existing evaluation is E.9.DA. If a campaign needs a different DRR adequacy coordinate, A.19.ECS can test whether that coordinate belongs inside E.9.DA, another evaluation, or no current FPF pattern.
Show, FPF Pillar adequacy. The evaluated object is FPF as a corpus or release candidate. E.2 gives the Pillars; E.2.DA is the evaluation. A.19.ECS explains why E.2.DA needs evaluated object, use, eligibility, coordinates, evidence loci, stop meanings, and neighbour governing relations rather than a Pillar essay.
Show, name improvement. The evaluated object is a durable term candidate. F.18 already supplies a grouped lexical quality vector: SemanticFidelity, CognitiveErgonomics, MorphologicalActionFit, and AliasRisk, plus NQD discipline over candidate names. A.19.ECS treats F.18 as an existing local evaluation for naming, not as a reason to build another one.
Show, no evaluation yet. A team says “make this onboarding method better” but cannot say better for whom, by what values, or with what stop. A.19.ECS opens before E.23: it names evaluated object kind, user, use, contrast cases, candidate characteristics, scales, floors, missingness, protected trade-offs, and neighbour governing relations. Only then can E.22 frame an improvement-oriented quality evaluation and E.23 improve the method.
A.19.ECS:5.1 - Bias-Annotation
A.19.ECS corrects score-first bias. Teams often begin improvement by choosing convenient scores, visible dashboards, or familiar criteria. The pattern starts instead from evaluated object kind, declared use, contrast cases, characteristic slots, scale bindings, value meanings, evidence basis, missingness, protected trade-offs, and stop conditions.
It also corrects evaluation-reuse bias. A reusable evaluation is useful only when it fits the object kind and use. If the existing evaluation already fits, use it; if it does not, construct or repair the evaluation characteristic space before starting an improvement loop.
A.19.ECS:6 - Conformance checklist
| Check | Requirement | Why |
|---|---|---|
CC-A19ECS-1 | An evaluation characteristic-space specification SHALL name evaluated object kind, use scope, reader scope, and qualification window. | Prevents context-free quality claims. |
CC-A19ECS-2 | It SHALL include admissible, below-floor, and outside-declared-object-kind boundary contrast cases. | Tests evaluated-object-kind discrimination. |
CC-A19ECS-3 | Each coordinate SHALL bind one characteristic to one scale, including for an ordinal content evaluation. State separately whether the value is a measurement claim or a content evaluation. | Preserves A.17/A.18/C.16/A.19 discipline. |
CC-A19ECS-4 | Each coordinate SHALL state value meanings, polarity or no-simple-direction value rule, evidence rule, and missingness rule. | Makes values replayable. |
CC-A19ECS-5 | The specification SHALL state floor, exceptional, status, stop, and reopen meanings for the declared use. | Lets improvement stop locally without claiming final perfection. |
CC-A19ECS-6 | Protected trade-offs SHALL be named when improving visible coordinates can harm another live value. | Blocks Goodhart-style improvement. |
CC-A19ECS-7 | The specification SHALL not average ordinal coordinates or turn undeclared coordinates into hidden pass, waiver, or failure. | Preserves non-scalar comparison. |
CC-A19ECS-8 | Wrong-kind objects SHALL return to evaluation selection before opening, or receive an explicit object-kind-fit defect/value when the evaluation has already been invoked. | Keeps the declared coordinate table complete after invocation and prevents false low scores before the suitable evaluation is selected. |
CC-A19ECS-9 | If made reusable beyond one local use, the evaluation characteristic-space specification SHALL make the minimum items in A.19.ECS:4.3 recoverable by value. If the selected publication form is an FPF pattern, E.8 also applies to that publication form. | Prevents underspecified evaluations. |
CC-A19ECS-10 | If the evaluation itself changes, justify its adoption or trial through §4.5 and name the comparability effect on earlier results. Keep known-case development, independent adoption evidence and full cost distinguishable. | Prevents silent value drift and improvement claims based only on the new evaluator. |
CC-A19ECS-11 | The evaluation characteristic-space specification SHALL state any evidence, assurance, gate, work, decision, publication, naming, measurement, Q-Bundle, OEE/NQD, mathematical-lens, or related claim as an exact subject assertion or named relation by value when that claim is being made. The coordinate Solution carries the evaluation construction itself; a subject-pattern locator, reference boilerplate, architecture-placement rationale, and neighboring content stay in relations, rationale, source-basis, or decision-rationale material unless they change a coordinate. | Prevents an evaluation from becoming a second ontology or reference boilerplate. |
CC-A19ECS-12 | A reusable evaluation characteristic-space specification SHALL state what would lower, reopen, or retire the evaluation: missing contrast case, changed use, changed source-use relation or source-currentness status, hidden trade-off loss, or corrected neighbouring-pattern claim assignment. | Makes high-value evaluation claims falsifiable instead of permanent praise. |
CC-A19ECS-13 | A reusable evaluation characteristic-space specification SHALL define the result-row shape and require a short rationale for every coordinate value. | Prevents prose impressions and two-column tables from being mistaken for evaluation results. |
CC-A19ECS-14 | It SHALL define the evaluation evidence basis and any coordinate-specific evidence payload needed for source-currentness, comparator, corpus-projection, worked-case, retrieval, or external-currentness claims. Apply the declared missingness rule when evidence is missing or unchecked. Lower a value for that absence only when the coordinate’s value rule evaluates evidence adequacy or otherwise makes that absence a lower value; do not infer a low property value merely from an unknown value. | Makes values replayable while preserving missingness and the declared coordinate coverage. |
CC-A19ECS-15 | It SHALL publish calibration points for common adjacent-value disagreements whenever the evaluation is expected to be reused by different evaluators. | Keeps 3, 4, and 5 from drifting into evaluator temperament. |
CC-A19ECS-16 | It SHALL declare where result evidence, corpus-projection evidence, retrieval evidence, comparator evidence, currentness evidence, and quality-status evidence live. These payloads SHALL stay in the evaluation result, evidence basis, projection carrier, or selected publication carrier unless the evaluated object itself is that carrier. If the payload implies a user-facing action for another evaluated object, publish that move or boundary, not the carrier proof. This is an evaluation-payload placement rule, not a lexical ban: evidence-use payloads do not enter live evaluated-object text merely because they are true or useful to authors or evaluators. | Prevents evaluation evidence from leaking into the evaluated object’s method or live text. |
CC-A19ECS-17 | If construction or repair changes coordinate wording or evaluation wording, the specification SHALL require a pre/post kind-restoration check for evaluated object kind, characteristic kind, relation or claim kind, current ontic slot, relation position, use relation, admissible use, and scope, plus the exact defining or constraining ClaimGraph when another pattern description contains it. | Prevents coordinate cleanup from changing what the evaluation evaluates. |
A.19.ECS:7 - Common Anti-Patterns and How to Avoid Them
| Anti-pattern | Symptom | Repair |
|---|---|---|
| Scale set from air. | Coordinates appear because they are familiar. | Rebuild from evaluated object kind, use, contrast cases, failure modes, domain tradition, first principles, and current source-use relation. |
| Wrong-kind object forced through the table. | Objects outside the declared kind are either scored as weak members of that kind or silently exempted from declared coordinates. | Add an object-kind-fit rule and boundary cases: before opening, return to a suitable evaluation; after invocation, record an explicit object-kind-fit defect/value or repair status. |
| Checklist masquerading as characteristic space. | A list of tasks is treated as coordinates. | Convert each task row to an evaluated EntityOfConcern property with a characteristic, scale, value meaning, and evidence rule, or move it to work planning. |
| One total quality score. | Several ordinal values are averaged. | Use coordinates, statuses, dominance or comparison rule, and protected trade-offs; do not scalarize unless an neighboring pattern governing the claim explicitly declares the operation. |
| Improvement without floor. | A loop continues because more change is possible. | State floor, exceptional meaning, stop condition, and reopen condition. |
| Hidden value drift. | The evaluation changes while old evaluations are compared as if nothing changed. | Version the evaluation and state comparability, bridge, or retirement. |
| Evaluation theft. | The new evaluation starts asserting evidence, assurance, gate, work, decision, or publication truth without the corresponding predicate and case facts. | State each neighboring subject assertion under its exact predicate or constraint and leave only the value evaluation here. |
| Result prose as evaluation. | An evaluator returns a narrative, two-column table, checklist count, or value list without evidence basis and short rationales. | Define the result-row shape, require short rationales and evidence basis, and apply the declared missingness rule to each coordinate whose needed evidence is missing or unchecked. |
| Evidence basis as evaluated-object method. | Corpus projection, retrieval, currentness, comparator, monolith-parity, quality-status evidence, or author or reviewer turn correspondence is written in the evaluated object as if it were what the evaluated-object user does. | Move the evidence to the evaluation result, evidence basis, projection carrier, or selected publication carrier; keep only the user action or boundary that the evidence justifies. |
| Coordinate wording as ontology change. | A coordinate or repair name sounds cleaner, but changes the evaluated object kind, characteristic kind, relation or claim kind, admissible use, or scope. | Treat it as a changed evaluation decision, recover the pre/post kind relation, and repair or reopen the evaluation rather than accepting lexical cleanup. |
A.19.ECS:8 - Consequences
A conforming A.19.ECS result lets E.22 ask a useful improvement-oriented quality-evaluation question and lets E.23 run a repeated improvement loop without inventing values during the loop. It also gives object-specific evaluation patterns such as E.21, E.9.DA, E.2.DA, and F.18 a common construction shape: evaluated object kind, use, contrast cases, coordinates, value meanings, evidence basis, result-row shape, calibration points, coordinate-specific payloads, protected trade-offs, status meanings, and local stop or reopen condition.
The cost is intentional. A reusable evaluation is heavier than a local checklist, because it must prevent wrong-kind use, hidden value drift, proxy-for-value substitution, neighbour theft, and false stop claims. When a local rubric is enough, keep the rubric local. When reuse is needed, carry the evaluation by value.
A.19.ECS:9 - Rationale
Improvement cannot be better than its evaluation. A loop that changes an object version without a declared characteristic space can only produce activity, persuasion, or evaluator preference. An evaluation that lists scales without evaluated-object-kind discrimination, floor, evidence, missingness, trade-offs, and stop meanings cannot guide improvement safely.
Placing this method under A.19 keeps the ontology clean. A.19 governs the structure of CharacteristicSpace; A.19.ECS governs the construction method for evaluations of declared EntityOfConcern kinds and uses. A.19.ECS governs the selected characteristics, scales, coordinate construction, and evaluation-use boundaries of the evaluation characteristic space, not its publication or record form. An FPF pattern is only one possible publication form when the evaluation belongs in FPF; a local rubric, standard, table, or project rule is enough when the use is local. E.23 stays a universal loop method because it does not need to know how every domain chooses its scales. Domain and FPF-specific evaluations such as E.21, E.9.DA, E.2.DA, and F.18 keep coordinate choices inside those evaluations.
A.19.ECS:10 - SoTA-Echoing
| Claim | Current practice line | Adoption in A.19.ECS | Boundary |
|---|---|---|---|
| Evaluation artifacts must declare intended use, object, criteria, and missingness before their values are useful. | Current reporting anchors: BenchmarkCards/EvalCards practice for evaluation-card structure, model-card lineage for intended-use and performance-characteristic reporting, and HELM/VHELM/AHELM-style evaluation suites for scenario, metric, raw-result, and modality-extension transparency. | A.19.ECS starts from evaluated object kind, use scope, contrast cases, coordinate meanings, evidence rule, and missingness rule. | It is not a benchmark harness, automated judge, or publication format by itself. |
| Multicriteria evaluation needs preserved dimensions and protected trade-offs. | Current QD overview: A survey on Quality-Diversity optimization: Approaches, applications, and challenges, Swarm and Evolutionary Computation 100:102240 (2026); retained design lineage: MCDA and value-focused thinking for criterion separation and trade-off visibility. | The pattern requires coordinate values, polarity or no-simple-direction value rule, protected trade-offs, status meanings, and stop or reopen conditions. | Scalarization belongs only to an neighboring pattern governing the claim or explicitly declared local method. |
| Improvement concern can damage the intended value when the evaluation is a weak proxy. | Current proxy-risk anchors: Goodhart's Law in Reinforcement Learning (ICLR 2024) and current catastrophic-Goodhart reward-misspecification work (NeurIPS 2024); retained lineage: Goodhart taxonomy. | A.19.ECS requires evidence rules, missingness rules, protected trade-offs, and lowering/reopen conditions before a loop can treat a value as improved. | It is not an anti-measurement rule; it makes the measurement or ordinal evaluation explicit enough to be challenged. |
| OEE and NQD work keeps the quality side distinct from novelty, diversity, archive, pool, and selected-set semantics. | Current QD, OEE, and NQD neighbour basis: quality-diversity work evaluates quality together with novelty and diversity, while archive and front are separate relations. Use C.17 for novelty and diversity retention, C.18 for archive and front relations, C.19 for pool treatment, G.5 for selected-set result declaration, G.9 for parity, and G.11 for currentness and refresh. When audience availability is current, use E.17 for a source-backed publication face and return to source and E.24.PUB for the publication occurrence, form, carrier, audience, bounded use, and availability. | An evaluation may supply Q values. It does not thereby establish neighboring search, selection, retention, currentness, or publication claims. Examples include novelty, diversity, archive, front, pool, selected-set, parity, refresh, and publication claims; apply the named definitions and tests only when the corresponding claim is current. | A.19.ECS constructs an evaluation U.CharacteristicSpace; using it neither performs nor establishes OEE or NQD generation, selection, archive, publication, parity, or refresh. |
| Evaluator quality must be connected to the decisions it improves. | Kim et al., Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization (2025), studies reward-model evaluation in bounded mathematical tasks. HCD.18/.19 and ME.14 supply developed FPF domain treatments of criterion error, support conditions and comparative practical worth. | Adapt: §4.5 compares missed defects, admissible difficult cases, proposed repairs and cost. This cross-domain construction requires its own subject grounds; the domain methods remain examples, not Core authority. | Reward-model correlation, evaluator agreement and a better specification do not establish general decision usefulness. Reopen when a target use cannot obtain the independent basis the comparison needs. |
| Adaptive reuse can turn adoption cases into tuning material. | Dwork et al., Generalization in Adaptive Data Analysis and Holdout Reuse (2015), is statistical lineage; Iacob et al., The Red Queen Gödel Machine (2026), is current bounded research on evolving evaluation. | Adapt: separate known-case development from independent adoption, preserve scope and expose cost in §4.5. | Neither statistical holdout guarantees nor benchmark improvement transfers automatically to qualitative framework evaluation. Reopen if leakage, a new population or comparison burden defeats the intended use. |
A.19.ECS:11 - Relations
| Pattern | Relation |
|---|---|
A.19 | Defines CharacteristicSpace. A.19.ECS gives the method for constructing one evaluation CharacteristicSpace for an evaluated object. |
A.17, A.18, C.16 | Govern characteristics, scales, scale values, coordinates, measures, units, measurement admissibility, and scale lawfulness. A.19.ECS uses them by reference for each slot. |
C.25 | Governs Q-Bundle normal form for composite engineering quality families. A.19.ECS may select or repair the characteristic-space part before a Q-Bundle endpoint is used. |
E.22 | Frames one improvement-oriented quality-evaluation question after an evaluation is declared. A.19.ECS constructs the missing or inadequate evaluation. |
E.23 | Governs repeated improvement after evaluated object version and evaluation are declared. A.19.ECS constructs a missing evaluation and supplies the adoption comparison when that evaluation changes. |
C.11.DUA | Supports choosing attainable evaluation effort from the useful decision it can change and the full cost of obtaining and using its result. |
E.21 | Existing evaluation for one FPF pattern version. A.19.ECS explains the construction shape but does not replace E.21. |
E.9.DA | Existing evaluation for one DRR decision-adequacy claim. A.19.ECS does not replace it. |
E.2.DA | Existing evaluation for FPF-level Pillar adequacy. A.19.ECS explains why it must publish evaluated object, coordinates, values, evidence loci, status, and stop meanings. |
F.18 | Existing naming discipline with a grouped lexical quality vector. Use F.18 for durable term and name improvement. |
C.16.P, C.16.Q, E.10, A.6.P, C.2.P | Repair overloaded characteristic/scale/score, quality, lexical, relation, and source-use wording before it becomes a coordinate or status value. |
C.18, C.19, G.5, G.9, G.11 | Govern OEE/NQD novelty, diversity, archive, pool, selected-set, parity, and refresh semantics. An evaluation may supply Q values, but it does not govern the rest of OEE/NQD. |
C.29 | Governs mathematical-lens use when a mathematical structure is used to define or justify coordinates. |
A.10, B.3, A.20, A.21, A.15 | Govern evidence, assurance, local CV, gates, and work when an evaluation result is reused for those claims. |