STR.7:4.4 - Design the smallest experiment that can supply the useful answer
Start from the needed observation and work backwards to what must actually be offered, used or performed. Choose a form that preserves those conditions with the least full burden. A minimum viable test is the smallest sufficient way to obtain that answer, not necessarily a smaller version of the final product. These examples distinguish different questions, not successive maturity levels:
| Question the result must answer | A possible test construction | What its result can support |
|---|---|---|
| Will these people take the offered next step? | Show a comprehensible offer or demonstration and provide an observable opportunity to accept or decline that step. | A response to that offer under those conditions. Viewing or joining a waiting list alone does not establish paid or continued use. |
| Will they use or pay for the proposed service under these terms? | Provide a limited real service, using competent manual work where that is permitted, and observe use, payment and delivery effort. | The service with that actual support. Human delivery does not establish the cost or performance of a future automated arrangement. |
| Can the proposed technical mechanism perform the required work? | Exercise that mechanism under the relevant conditions, or use qualified evidence that answers the same claim. | Performance within the tested or qualified conditions. A human substituting for the mechanism cannot establish that mechanism’s performance. |
Remove a feature or substitute a simpler component only while the required observation and its interpretation remain possible. Retain the work needed to collect the observation, protect participants and close the activity. Explain what participants will receive and who may handle their information; do not sell simulated performance as a delivered capability.
Use the appropriate domain design to connect observations to the bounded claim. State the participants or systems exposed, contribution being tried, conditions of use, observations, interpretation and limits. Specify comparison or control conditions when the inference requires them; merely naming a pilot does not establish a causal effect.
When the decision specifically needs the effect of changing an offer or feature, one possible design is a concurrent randomized comparison. Construct it as follows, with competent domain and statistical support where the inference needs it:
- State the change and comparator, the people or systems to which the answer must apply, the outcome and any material harms. Choose a meaningful difference and observation period from the receiving decision. Keep the other conditions compatible with the intended comparison.
- Define the units to be assigned, then allocate eligible units to the variants by a random procedure under stated probabilities before exposure. Keep a unit’s assignment consistent unless the design explicitly handles switching. Repeated visits by one person are not automatically independent units; interaction between participants or competition for shared resources can require a different design. Sending each offer to half the recipients does not establish random assignment.
- Specify the sample size needed for decision-relevant precision, whether it is attainable, the outcome calculation, missing-observation treatment and analysis or stopping rule before inspecting the effects. Retain assignment, actual exposure and outcome evidence for both variants, including departures from the plan. Check whether observed group sizes and missingness are compatible with the allocation and collection design; resolve a material unexplained discrepancy before relying on the effect. Judge the resulting difference with its uncertainty and supported population, configuration and period.
This construction starts a controlled comparison; it does not make every pilot an A/B test or supply a qualified causal estimate by itself. C.28 governs that stronger use. Use another qualified design or an adequate existing result when it answers the question better, and return an unattainable inference rather than exposing participants to an uninformative test.
Before dependent activity, identify who may authorize participation, access, spending and changes to existing provision. Set the resource ceiling, duration or stopping event, exception response and withdrawal or restoration conditions. The ceiling includes the work needed to close the experiment responsibly.
Agree what a positive, negative or unresolved result would support. A successful local delivery may support a larger bounded comparison, not a general demand forecast. A failed result may reject this configuration without disproving every possible direction.
Keep missing conditions explicit. If the design cannot be completed until permitted use or support is established, return that bounded gap and preserve independent work. A proposed experiment is not an available action merely because its question is valuable.