Choose a research brief, inspect its sources, and adapt the plan for your own work. Read, edit and export without an account.
Choose inputs, comparisons and metrics before trusting a score. Read the collection guide →
How misleading can a nominal 95% success-rate interval be when only twenty independent tasks are observed?
Do evaluation harnesses agree on the same frozen predictions once normalization, aggregation and failure handling are made explicit?
Can another operator reconstruct one bounded agent-evaluation attempt from its task, environment, actions, failures and scoring rule?
Platform-authored guidance for your own work. Each task states a comparison to investigate and the limits of what its results could establish.
For 20 independent trials with success probability 0.05, how often do nominal 95% Wald and Wilson intervals contain the true probability?
DEFINED INPUTSBinomial model: n = 20, exact p = 1/20, nominal coverage 19/20.
COMPARISON TO INSPECTSum binomial probability over each of the 21 counts whose interval contains p. Full enumeration does not imply exact floating-point arithmetic.
For five values near one billion, how do floating-point variance formulas compare with the exact population variance?
DEFINED INPUTSThe five integers are 999999998, 999999999, 1000000000, 1000000001 and 1000000002.
COMPARISON TO INSPECTThe exact mean is 1000000000 and centered squared offsets sum to 10, giving population variance 2.
Do two graph algorithms produce the specified reference distances and distinguish a negative edge from a reachable negative cycle?
DEFINED INPUTSPositive graph: A→B:2, A→C:5, B→C:1, B→D:4, C→D:1; source A.
COMPARISON TO INSPECTHand-derived positive distances are A:0, B:2, C:3, D:4. Both methods should be compared with this map.