Experiments: Matched-Budget Runs Over a Seeded Held-Out Split
What an experiment is
Section titled “What an experiment is”The experience store indexes settled runs and uh observatory compare compares
two arms of a set of runs that already exist. An experiment is the thing
that produces comparable runs on purpose: a task set with executable checks,
partitioned into a search and a held_out split, executed under two or more
configurations (arms) at a matched budget, and reported honestly against the
baseline of simply repeating the cheaper configuration.
An experiment is declared in .harness/experiments/<id>.yaml and validated
against the strict uh.experiment.v0 schema (src/schema/experiment.ts). Its
tasks are mission ids under .harness/missions/, and every task mission must
declare verification.required_checks — a task with no executable check cannot
be scored.
schema_version: uh.experiment.v0id: denial-tonetitle: Guard denial tonetasks: [mission-a, mission-b, mission-c, mission-d]split: seed: 7 held_out_fraction: 0.34 # defaultarms: - id: terse template: balanced - id: explanatory template: balanced runtime_config_overrides: denial_tone: explanatory attempts_per_task: 1 # defaultbudget: max_runs: 40 max_total_cost_usd: 25baseline: arm: terse parallel_attempts: 1 # defaultThe invariants
Section titled “The invariants”- The evaluator lives outside the loop. A run is scored by the task’s own
verification.required_checksand its settled run record, never by the model that produced it: pass rate comes fromstatus, denials fromdenials, the guard classes from thetool-guard.log, and cost from the priced run record. - Held-out acceptance is not optional. The split partitions the tasks, and
the report always presents each partition separately. A gain that appears only
on
searchtasks is not a gain; published harness-evolution work shows search gains vanishing on held-out tasks, so the report never pools the two. - The budget is matched. Arms are interleaved at every attempt level
(
planExperiment), so a budget stop trims each arm by the same amount and can never favour one arm. When a budget line is reached the remaining plan entries are recorded as skipped with reasonbudget, and skipped rows are never counted as evidence. - Optimizer tokens are counted. Token totals and cost are read from the run
record the experience store already builds. Unknown cost is never treated as
zero: it stays
unknownin the report, and an unpriced run contributes nothing to the cost budget (so it can neither look free nor silently spend past the cap). - Determinism. A seeded split is a pure function of
seedandheld_out_fraction(mulberry32), so a seed always names the same partition and the seed is part of every output. An explicitsplit.held_outlist is used verbatim.
File layout
Section titled “File layout”.harness/experiments/ <id>.yaml # the spec (uh.experiment.v0) <id>/ plan.json # seed, split sizes, and the ordered {task, arm, attempt, split} plan runs.ndjson # one row per plan entry: the settled record, or a budget skip report.json # per-split per-arm summary, arm comparisons, baseline repeats report.md # the same report as markdownrunExperiment writes a experiment.json ({ id, arm, split }) into each
run’s directory under .harness/missions/<task>/runs/<run_id>/, the same way a
session template records session-template.json. indexRuns reads it into the
run record’s optional experiment field, so an operator can group a mission’s
runs by experiment and arm.
Module and CLI surface
Section titled “Module and CLI surface”src/harness/experiment.ts is pure except for the injected runner:
splitTasks(spec) and planExperiment(spec) are deterministic and
side-effect-free, and runExperiment(root, spec, { runner, now }) owns only the
budget loop, the persistence, and the tagging. The runner has the shape of the
per-runtime runtimeRunner seam in harness/run-all.ts: tests inject a fake and
never start a model runtime. summarizeExperiment(runs, spec) groups rows by
split and arm, calls the same compareArms the observatory uses for every arm
pair within a split, and adds the baseline arm’s best-of-n and
attempts-to-match lines.
uh experiment plan <id> # seed, split sizes, arm-interleaved planuh experiment run <id> [--json] # execute and persist plan/runs/reportuh experiment report <id> [--json] # summarize persisted runs against the baselineHuman output prints the seed, the split sizes, each arm per split with its
Wilson interval, the mean denials and the guard_tamper / containment_escape
stop counts, the verdict sentence, and the plain-repeats line.