Evaluation
What changed, compared with what, under which conditions? A useful evaluation makes the baseline, costs and uncertainty inspectable.
Synthetic evaluation example
The opponent mix can change the result.
Assume a fixed poker strategy returns +8 bb/100 against population A and −2 bb/100 against B, before costs. At an 80/20 mix its expectation is +6 bb/100. At a 20/80 mix it is zero.
These are assumed values, not measured win rates. The example shows why the evaluation population belongs beside the result. It does not establish strategy performance.
Inspect the example and cost assumptionsWhat the report must show
| Measure | Basis | Record |
|---|---|---|
| Net return | Game or market, outcome probabilities, payout and complete cost. | Value per unit staked, period, sample and assumptions. |
| Strategy difference | Comparable actions, information, rake and opponent assumptions. | Difference from a defined baseline, with uncertainty. |
| Available volume | Eligibility, accepted size, limits, expiry and capacity. | Volume supported by the rules and records. |
| Loss exposure | Outcome distribution, capital, dependence and stop conditions. | Plausible losses, variance and review triggers. |
| Model stability | Held-out records, later periods and changed inputs. | Evaluation output and conditions that reverse the conclusion. |
Match the test to the claim
A live result requires live observations. A retrospective result needs dated data and the information available at each decision. A simulation needs its model and opponent assumptions. Those statuses are distinct.
A fair comparison holds the stake basis, available information and cost treatment consistent. Excluded or rejected inputs and operating restrictions belong in the report, along with dataset and configuration versions.
Keep dependence and uncertainty visible
The same player, event or session can create dependence even when individual rows differ. Evaluation splits must account for those relationships and the intended future use.
A sensitivity test asks how a conclusion changes when assumptions move. It is not a statistical confidence interval. Failure slices and conditions that reverse the conclusion matter alongside the average result.