The things a practitioner checks before trusting a tool, stated plainly.
- How a winner is called
- A Bayesian posterior past 0.975, the same verdict on two consecutive reads, and enough data in each arm that a lucky handful cannot decide it. All three, or the test keeps running.
- Guardrail metrics
- Name a goal that must not suffer and a variation that lifts the primary while costing it is blocked rather than shipped. After promotion a holdback slice keeps seeing the control, so a win that reverses is caught rather than assumed.
- The belief ledger
- Every result is filed against the idea it tested, with the evidence attached, at one of six statuses: untested, promising, proven, disproven, exhausted, backfired. A tie is recorded rather than discarded, and beliefs are matched by meaning, so renaming a dead idea does not revive it.
- Signal isolation
- Each hypothesis is expressed as several variations with different phrasing, and execution patterns are tracked separately from the idea. A win across phrasings is the argument; a win in one is a good sentence.