Make an agent benchmark attempt auditable
Can another operator reconstruct one bounded agent-evaluation attempt from its task, environment, actions, failures and scoring rule?
Make this plan your own ↓Read, edit and export without an account. This is preparation; no run or result is claimed.
Make the question your own.
6 sources · Exact preparation context
- SWE-bench →
Preserve the upstream project name, repository, exact source commit and applicable contributor/notices; resolve upstream citation guidance for any later formal use.
Record revision: 1f31e75c09dbdc82d77006e8f81c4de720d9d47f1ea05f98d079adb3cbd5705c
- mini-SWE-agent →
Preserve the upstream project name, repository, exact source commit and applicable contributor/notices; resolve upstream citation guidance for any later formal use.
Record revision: d85f6be8b78c3e3e418f8db3071d410b0d164e590bc1d3d29d8db6f4c381766d
- AgentBench →
Preserve the upstream project name, repository, exact source commit and applicable contributor/notices; resolve upstream citation guidance for any later formal use.
Record revision: 2cb203b47e5f3887f6a369d5f3b01ba55b4b6e6a8c401787b5702b5f3d3d3680
- SciCode →
Preserve the upstream project name, repository, exact source commit and applicable contributor/notices; resolve upstream citation guidance for any later formal use.
Record revision: 2cd64e8968ece525554267c471f8d32d72f5eb9c05f6a30baca0de45fb3aae47
- EvalPlus →
Preserve the upstream project name, repository, exact source commit and applicable contributor/notices; resolve upstream citation guidance for any later formal use.
Record revision: b013666ce40ca755b776a6148550f33b59f063a952519bb1ee731363d4d8214d
- HumanEval →
Preserve the upstream project name, repository, exact source commit and applicable contributor/notices; resolve upstream citation guidance for any later formal use.
Record revision: 476aa19deba6efb2ef9b44698d560702b471d74b2327f7eb155c3ab6ca3c6147
Source links open the current listing. This plan retains the revisions above; listing review does not grant permission to copy or adapt source files.
Before carrying out this plan
- Benchmark payloads, issue text, reference code and model rights need their own checks.
- Agent-generated code and benchmark environments require a qualified isolated execution path; ordinary host execution or a local container is insufficient.
- Resolve SciCode's recorded route/dependency caveat and any moving data default before pinning an input.
Limits to keep in view
- None of these benchmarks or agents was executed by this collection.
- No success rate, independent replication, safe sandbox qualification or superiority claim is made.
Edit freely. Export a copy before reloading or leaving. Your text stays in this tab until you choose to copy, download or save it privately.
What to compare
Reconstruct a single attempt before comparing aggregate rates. Keep retries and unsuccessful attempts in the denominator specified by the protocol. Author actions and generated tests cannot manufacture a platform-controlled verdict.
Useful outputs
- One selected task and immutable data/code/environment identifiers with scoped rights records.
- An explicit allowed-tools/network policy, model/prompt configuration, attempt budget and immutable scoring rule.
- An attempt log preserving timeout, infrastructure failure, invalid output and scientific/task outcome separately.
Inputs and prerequisites
- Benchmark payloads, issue text, reference code and model rights need their own checks.
- Agent-generated code and benchmark environments require a qualified isolated execution path; ordinary host execution or a local container is insufficient.
- Resolve SciCode's recorded route/dependency caveat and any moving data default before pinning an input.
What this work would not establish
- None of these benchmarks or agents was executed by this collection.
- No success rate, independent replication, safe sandbox qualification or superiority claim is made.
Start with these sources.
SWE-bench
Upstream project repository
Software issue-resolution benchmark and evaluation tooling for studying patch assessment and reproducibility.
Investigate Can one explicitly licensed fixture demonstrate that patch application and test-result collection preserve the original issue/task identity?
mini-SWE-agent
Upstream project repository
Compact software engineering agent implementation for examining minimal agent loops and inspectable action traces.
Investigate Can a tiny offline task distinguish task success from a successful shell command in a recorded agent trajectory?
AgentBench
Upstream project repository
Multi-environment agent benchmark repository for investigating task adapters and evaluation consistency.
Investigate Can a single reviewed environment separate malformed agent actions, infrastructure errors and genuine task failures?
SciCode
Upstream project repository
Scientific coding benchmark repository linking problem formulations and evaluation logic for researcher-oriented model assessment.
Investigate Can one rights-reviewed scientific coding task distinguish a syntactically valid solution from numerical correctness under fixed tolerances?
EvalPlus
Upstream project repository
Code-generation evaluation tooling and expanded testing resources for studying test coverage and result reproducibility.
Investigate Can a hand-authored incorrect solution pass a weak fixture but fail a separately specified edge-case test?
HumanEval
Upstream project repository
Code-generation evaluation repository for examining functional correctness and pass-at-k estimation on programming tasks.
Investigate Does the pass-at-k aggregation match an independently enumerated finite set of synthetic candidate outcomes?