Evaluating agent runtimes: pass^k over pass@1

Agent benchmarks mostly answer the wrong question. pass@1 tells you whether a system can solve a task; operators need to know whether it will — every time, at what cost. RFC 0024 specifies agentd's evaluation harness around that operational framing.

The metric is pass^k: the probability that all k independent attempts succeed. Where pass@k rewards a lucky outlier, pass^k punishes variance — which is the property a CronJob that runs nightly actually depends on. Alongside it, every result carries cost: tokens, wall-clock, and tool-call counts, because a run that succeeds for twenty dollars is a different product than one that succeeds for twenty cents.

The suites are external and unmodified: MCP-Universe for tool-use breadth, τ²-bench for conversational tasks, BFCL for function-calling fidelity, SWE-bench Verified for software work, GAIA for general assistance, Terminal-Bench for shell competence. The harness treats the runtime × model pair as the unit under test — the same instruction, the same MCP surface, different intelligence — which is the comparison a minimal runtime invites: if agentd's thesis is "the runtime should add nothing but supervision," the benchmark's job is to catch it adding failure.

Phase 0 is runnable today in the repository's bench/ directory. The RFC is honest about what remains: full-suite integration, statistical treatment of flaky tasks, and the publication pipeline. It earns its place in this research zone as a position statement: reliability-per-dollar is the number agent infrastructure should be judged on.