An exact small-sample permutation reference
Enumerate every balanced assignment for two groups of four observations and compare the exact two-sided randomization p-value with a sampled estimate.
Proposed original example. Source release and any execution are separate steps.
Can a Monte Carlo permutation implementation be checked against a fully enumerated finite reference?
Evaluate all 70 assignments using Fraction arithmetic and an explicitly inclusive tie rule; record the exchangeability assumption separately.
What you could produce
- A scoped observations JSON record and a comparison with the stated reference.
Before you use it
- Python 3.13 standard library
Limits to keep in view
- No research code was executed by the preparation tool.
- Author output cannot issue an independent scientific-verification result.
Source and permission context
Original local preparation by the Executable Science seed collection; upstream API references remain separately attributed.
Rights need review. Review the scope and upstream conditions before reuse.
Still unresolved
- Proposed local-draft licenses: original code MIT, explanations CC-BY-4.0, synthetic numeric data CC0-1.0; publication/disclosure approval remains separate.
Put this resource to work.
Put uncertainty around a small agent-success benchmark
How misleading can a nominal 95% success-rate interval be when only twenty independent tasks are observed?
Open the brief RESEARCH BRIEF · 4 SOURCESCheck the false-positive cost of trying many variants
How does testing twenty null variants change the chance of declaring at least one apparent improvement?
Open the brief