How we work

Evaluation

What changed, compared with what, under which conditions? A useful evaluation makes the baseline, costs and uncertainty inspectable.

Synthetic evaluation example

The opponent mix can change the result.

Assume a fixed poker strategy returns +8 bb/100 against population A and −2 bb/100 against B, before costs. At an 80/20 mix its expectation is +6 bb/100. At a 20/80 mix it is zero.

These are assumed values, not measured win rates. The example shows why the evaluation population belongs beside the result. It does not establish strategy performance.

Inspect the example and cost assumptions

What the report must show

Criteria for assessing return, volume, loss and model stability; not a report of client results.
MeasureBasisRecord
Net returnGame or market, outcome probabilities, payout and complete cost.Value per unit staked, period, sample and assumptions.
Strategy differenceComparable actions, information, rake and opponent assumptions.Difference from a defined baseline, with uncertainty.
Available volumeEligibility, accepted size, limits, expiry and capacity.Volume supported by the rules and records.
Loss exposureOutcome distribution, capital, dependence and stop conditions.Plausible losses, variance and review triggers.
Model stabilityHeld-out records, later periods and changed inputs.Evaluation output and conditions that reverse the conclusion.

Match the test to the claim

A live result requires live observations. A retrospective result needs dated data and the information available at each decision. A simulation needs its model and opponent assumptions. Those statuses are distinct.

A fair comparison holds the stake basis, available information and cost treatment consistent. Excluded or rejected inputs and operating restrictions belong in the report, along with dataset and configuration versions.

Keep dependence and uncertainty visible

The same player, event or session can create dependence even when individual rows differ. Evaluation splits must account for those relationships and the intended future use.

A sensitivity test asks how a conclusion changes when assumptions move. It is not a statistical confidence interval. Failure slices and conditions that reverse the conclusion matter alongside the average result.