MMP.14:4.2 - Generate the comparison the question requires

Compare quantities with the same meaning. Replicated physical states are not yet rounded, censored or selected records. Carry them through MMP.7’s recording law.

For a posterior predictive comparison, obtain joint parameter draws from MMP.13 and generate records conditionally: [ \theta^{(s)}\sim\pi_M(\theta\mid y,x),\qquad y^{\mathrm{rep},(s)}\sim p_M(y^{\mathrm{rep}}\mid x,\theta^{(s)}). ] Here (M) names the model and (x) the retained input and recording conditions. Compare (D(y)) with (D(y^{\mathrm{rep},(s)})). If the discrepancy depends on parameters, calculate (D(y,\theta^{(s)})) and (D(y^{\mathrm{rep},(s)},\theta^{(s)})) at the same draw.

Decide what repeats. New observations for existing groups with their inferred effects differ from new groups with regenerated effects. Retain or regenerate effects according to the questioned prediction. Do not independently redraw an effect that should be common to an entire batch.

A non-Bayesian comparison can use a specified parameter value, an exact conditional reference distribution that eliminates a nuisance parameter, or a fitted-model simulation. State which is used. When the reference concerns a statistic of a fitted procedure, reproduce the fitting step on each simulated dataset; a simulation that holds its fitted coefficients fixed generally answers a different question. A parametric bootstrap may approximate a reference distribution, not make its calibration exact.

For a held-out comparison, construct the forecast without using the records being predicted to fit, tune or select that forecast. Keep preprocessing inside the corresponding training operation. Choose the withheld unit and allowed information for the receiving prediction: new observations in an existing group, a whole new group, or a future block with a specified horizon. A convenient random split does not by itself establish future or new-group performance. An alternative split needs an argument that its bias and variability suffice for the intended conclusion.

Write a held-out predictive law as (Q_M(y_H\mid y_T,x)), where (T) is the available training information and (H) the withheld portion. Compare the models on the same (H), target and scoring convention. Use a joint block prediction when the question depends on within-block dependence. Pointwise scores can still answer a declared marginal prediction question; they do not test all joint behavior.

Exact enumeration or algebra may replace simulation. If a deterministic prediction and an observation each have established error bounds, compare their admissible ranges. Disjoint ranges expose an incompatibility under those bounds; overlapping ranges do not prove the model.