Skip to content

Ultimate Harness — Roadmap

Source of truth for issue state is Linear; GitHub mirrors through the native integration. This document describes release objectives and unresolved engineering work. Implemented changes are recorded in the changelog.

v0.11.0 — Integrated execution lifecycle

Section titled “v0.11.0 — Integrated execution lifecycle”

In development; not release-ready. Existing adapter, supervision, recovery, resource admission, verification, and review implementations must work together under failure. Passing isolated tests or adding an adapter does not establish end-to-end acceptance.

  • Complete packet persistence: uh mission put installs validated packets into .harness/missions/<id>/mission.yaml atomically with audit event logging, and uh mission check validates packet write roots and grounding claims. Persisting full dynamic schemas directly from coordinator propose paths remains open. Status: in progress.
  • Prerequisite admission: dispatch must consume valid required outputs, not merely observe that prerequisite processes exited. Missing or invalid reports must prevent dependent paid execution. Status: open.
  • Automatic monitoring and reconnect: uh ps, uh wait, and uh report provide model-free observation and synchronous settlement tracking; live run digests (uh.run-digest.v0) update on heartbeat. Controller-loss detection and duplicate-controller prevention are hardened; unattended reconnect remains open. Status: in progress.
  • Mid-run steering: uh steer messages active runs by requesting the controller to stop with steered and resume with the message injected without consuming restart budgets; uh note and uh ledger track intervention records. Steering an orchestrator stops its session and leaves the workers it launched orphaned; steer an orchestrator only between its steps until that is fixed. Status: shipped, with that defect open.
  • Queued launches: uh queue run launches missions in order under an orchestrator cap and a free-memory floor, settles each from its run record, and resumes from .harness/queue/<id>/state.json after a crash. Status: shipped; first live queue pending.
  • Gated landing: uh land lands verified, reviewed worker branches as one commit after the full checks, an attribution scan and the build, or restores the target exactly. Status: shipped; first live land pending.
  • Hidden grading: uh mission run --post-checks runs operator checks the agent never sees and fails the run when one fails. Status: shipped.
  • Coordinator context: limit inherited tools, integrations and task context according to explicit policy. Measure total consumption across coordinator, workers and retries; prompt length alone is not a sufficient measure. Status: open.
  • Resolve Claude context-suffix model identity without weakening route checks. Case and provider-prefix variants are reconciled; suffix variants are not. Status: open.
  • Distinguish native permission refusal from UH hook denial: native refusals without hook invocation are tracked as native refusals, consuming denial budgets without disarming the guard or triggering missing-hook stops. Status: shipped.
  • Validate Claude coordinator isolation against inherited settings, memory, skills and caller-supplied CLI arguments. Tool/MCP flags alone are not full isolation. Status: open.
  • Replace repeated scans of accumulated Claude events with bounded incremental accounting: uh report renders from uh.run-digest.v0 and reads event tails; bounded event accounting is active for live digests. Status: in progress.
  • Define explicit consumption/context policy. Live usage fields report measurements and unknown-cost admission policies manage wave entry; hard context caps remain open. Status: in progress.
  • Complete live team accounting and test stale run-index/control combinations in the Observatory. Verify actual presentation, not only projected JSON. Status: open.
  • Add stop-reason cost attribution without confusing runtime estimates with bills. Status: open.
  • Derive per-worker memory limits from declared subprocess requirements where supported, while retaining pre-launch refusal on unsupported platforms. Status: open.

Current JEV integration covers verification and independent-review collection. The progressive-decision specification also describes routing, scope-change and retry/stop flows.

  • Add validated mission decision policy and enforce confidence/provider requirements: DecisionPolicySchema and progressive semantic routing shipped; the opt-in shadow loop watchdog (runtime_config.loop_watchdog: shadow, off by default) evaluates retry/stop advisory receipts. Status: in progress.
  • Bound provider requests independently of subprocess deadlines. Status: open.
  • Validate the entire response envelope, including disabled-provider discrimination; distinguish transport failure, invalid JSON and malformed typed answers. Status: open.
  • Hash submitted evidence before consumer mutation; bound provider metadata and account for judgment usage without persisting raw prompts or responses. Status: open.
  • Define privacy and evidence-sufficiency rules for criterion descriptions and compact summaries. A judgment cannot establish facts omitted from its input. Status: open.
  • Keep deterministic failure and human authority dominant: failed required checks dominate as deterministic failures and zero judged criteria cannot yield pass. Status: shipped.
  • Extend independent review beyond one runtime/model per packet where multiple reviewers or model-family independence are required. Status: open.
  • Reconcile promotion policies with verified approver authority; a nonempty approved_by string is not authenticated identity. Status: open.
  • Complete credential scoping, isolated worker homes, deny-read protections and preflight verification: Tool Guard arms fail-closed, isolates write roots, denies agent/network clients, and traps guard_tamper and containment_escape. Status: in progress.
  • Extend declared-output checks beyond nonempty files, JSON and configured markers: declared outputs under gitignored paths are force-staged, and salvage commits strictly enforce write roots. Status: in progress.
  • Define consistent completion semantics across runtime results and worker reports: review-prepare captures worker runtime-final.txt, verification.yaml, and team records with hash pinning. Status: in progress.
  • Improve liveness diagnosis using pending tool-call structure as well as heartbeat freshness; an open terminal or recent event is not proof of forward progress. Status: open.

Exercise the connected paths below with explicit failure and recovery criteria. Keep reproducible definitions in the repository and execution artifacts in private local or CI storage. Do not ship local session history as release evidence.

Area Required behavior
Planning Research → specification → file/check plan → complete validated mission → sandbox and prompt hydration.
Dispatch Preflight and exact route → guarded execution → streamed artifacts → required-output validation; failure prevents dependent dispatch.
Supervision Startup, wall, stall, turn, denial, repetition and output limits preserve distinct causes and settle the owned process tree.
Recovery Cancellation targets one attempt; controller loss reconciles authoritative settlement; eligible saved-session resume retains work; deadline grace remains explicitly incomplete.
Teams Per-worker contracts, resource-bounded waves, isolated worktrees, integration and verification; unknown cost blocks further paid admission.
Verification Executable checks, output/TDD gates, hash-bound independent review, manual dispositions and promotion policy preserve deterministic failures and human authority.
Decisions Typed recommendations are consumed only within deterministic policy, with explicit unavailable, malformed and uncertain outcomes.
Observation Fresh, stale, unknown, failed and completed states remain distinct across native processes, teams, accounting, UI and reconnect.
Comparison Cross-adapter execution retains divergences and failures instead of treating unlike native behavior as equivalent.
Platforms Windows guardian, POSIX cleanup, sandbox backends, TUI and plugin boundaries are verified independently; unsupported behavior fails explicitly.

The acceptance registry is a mechanism-level subset, not the full release scope. Fixture checks do not establish live model behavior, and stale results do not establish the current revision. See acceptance procedures and verification.

The default no-evidence report is a registry view, not a portable evidence bundle.

v0.9.0 (current release, “Memory & adapter matrix” — Linear UH-131 / UH-136 / UH-137; GitHub PRs #204–#206 / #214 / #215 / #216): bundles everything merged to dev since v0.8.0. Ships a native Anthropic adapter (anthropic, status: experimental, #214) — the official, ToS-clean pay-per-token Messages-API path (ANTHROPIC_API_KEY env-only, blocked classification, graceful no-key checker, optional live-smoke CI job); Honcho memory operations (#215) — honcho_search / honcho_remember harness-side ops + per-mission runtime_config.honcho_memory opt-out on every Honcho-aware adapter; team-run dogfood fixes (#216) — UH-127 passed_partial verdict (no more false BLOCKED), UH-128 per-worker artifact-bleed fix, UH-129 integration_report_path default under .harness/missions/<id>/team/, UH-130 constraints-advisory warning; and Phase-0 DX hardening (#204–#206) — real uh --version from package.json, an opt-in PostHog telemetry primitive (unwired; UH-135 follow-up), adoption docs, a curated npm files: allowlist, and CI plugin gates. Will be published as @agenticengineeringagency/ultimate-harness@0.9.0 on npm latest (tagged v0.9.0 + plugin-v0.9.0 on main via the dev→main release PR). See CHANGELOG.md [0.9.0].

Epics 2–5 plus Epics 6–8 (live SSE tail + cancel, adapter capability routing + cost, SDD hardening) shipped and were released as v0.6.0 — published to npm as @agenticengineeringagency/ultimate-harness@0.6.0. v0.6.0 also corrected two v0.5.0 gaps: the auto-router uh mission run --auto (UH-101, claimed shipped but never built) and runtime.usage token capture (which the cost features assumed). The Epics 6–8 execution spec is specs/epics-6-7-8.md; per-slice detail is in the CHANGELOG [0.6.0]. v0.7.0: OpenRouter adapter ([#134]), vanilla pi adapter ([#135], shipped active), sandbox backend abstraction + directory backend ([#136]) and a container stub ([#137]), verify-then-promote auto-trigger ([#139]), capability-match enforcement ([#138], already shipped), and a shared adapter-artifact-helper refactor ([#133]). v0.8.0: OpenSandbox-gated container execution backend ([#155]/[#157]) with ADR + runbook ([#154]), and oh-my-pi graduated to active ([#156]); published as @agenticengineeringagency/ultimate-harness@0.8.0. v0.9.0 (current release, see Now above): native anthropic adapter (experimental, [#214]), Honcho memory operations + opt-out ([#215]), team-run dogfood fixes ([#216]), and Phase-0 DX hardening ([#204]–[#206]). Issue tracking: Linear is the canonical source of truth for the UH team; GitHub issues mirror Linear via the native 2-way integration (per-issue auto-creation + state/title/body sync). Cite Linear IDs (UH-NNN) in commits, PRs, and docs — the GitHub mirror picks up the back-link automatically. The v0.5.0/v0.6.0 epic records (UH-92..UH-111) and the v0.7.0/v0.8.0 milestone slices (UH-112..UH-125) were backfilled to Linear post-workspace-upgrade so the history matches the CHANGELOG.

Epic 5 — Hermes Dashboard plugin v2: multi-run history + replay + observability (UH-84)

Section titled “Epic 5 — Hermes Dashboard plugin v2: multi-run history + replay + observability (UH-84)”

Surface the per-run artifact backend (UH-82, shipped in v0.4.0-prep) in the dashboard UI. Today’s plugin shows one mission ↔ one “latest result”; v2 turns the dashboard into a triage surface for missions with dozens of runs.

Phase Issue Slice Size Status
Core UH-85 Recent runs pane on Mission detail S shipped — #98
Core UH-86 Per-run drilldown (?run_id= param) M shipped — #98
Core UH-88 Filter + search runs in the pane S shipped — #98
Later UH-89 Compare two runs side-by-side M shipped — #101
Later UH-87 Replay a run with same/modified overrides M shipped — #101
Optional UH-90 Run history retention policy S shipped — #100

Epic 5 shipped end-to-end across #98 + #100 + #101. Backend foundation shipped earlier in PR #96 (UH-82): runs/<run_id>/ per-run dirs, runs/index.json history, latest.json pointer, per-run artifact route honoring run_id, get_mission returns runs[]. TUI polish slices (UH-48 / UH-49 / UH-50 / UH-51) shipped in #99. Cut as v0.4.0 (#103 + #105 + #107); superseded by v0.5.0 and v0.6.0, all live on npm.

Ship a Hermes dashboard plugin that exposes UH end-to-end from the Hermes web UI — adapter health, mission list, mission run trigger, artifact drilldown, and theme integration. The plugin is a drop-in at ~/.hermes/plugins/uh/dashboard/: manifest.json + IIFE JS bundle (using window.__HERMES_PLUGIN_SDK__) + Python FastAPI router that shells out to the uh CLI. No daemon, no FFI, no Hermes fork.

Phase Issue Slice Size Status
Core UH-61 Plugin scaffold + dev workflow S shipped — #89
Core UH-62 Python ⇄ uh CLI FastAPI backend bridge M shipped — #89
Core UH-65 UH overview tab (adapters + missions + recent runs) M shipped — #89
Core UH-64 Mission run trigger + live event-stream tail M shipped — #89
Later UH-63 Mission drilldown viewers (prompt / final-message / diff / runtime-result) M shipped — #89
Later UH-66 Workflow profile viewer + verification report viewer M shipped — #89
Later UH-69 UH theme YAML + sessions:bottom slot cross-link S shipped — #89
Optional UH-67 New-mission wizard + workflow editor M shipped — #89
Optional UH-68 Plugin packaging, install runbook, and distribution repo S shipped — #89

All Core + Later + Optional slices shipped in #89 after 13 rounds of Codex adversarial review — 26+ documented correctness findings (race conditions, silent failure modes, security gaps, schema drift, response shape inconsistencies) all addressed before merge. See the Linear issues for per-slice detail; follow-ups UH-81 / UH-82 / UH-83 track the deferred items.

Distill the convergent learning from oh-my-claudecode and oh-my-codex into a UH-native mission shape: a single mission that fans out across multiple adapters in parallel as workers, each in its own worktree, with a leader/integrator role that merges their work and runs the existing verification pipeline over the integrated result. Extends UH-56 mission run-all from comparison to collaboration.

Phase Issue Slice Size Status
Core UH-71 Mission schema for team shape with adapter-bound workers S shipped — #88
Core UH-72 Per-worker worktree auto-isolation + leader merge / integration-report.md M shipped — #87 + #90
Core UH-73 Staged workflow profile family (plan → prd → exec → verify → fix) M shipped — #87

Standalone discipline-layer slices inspired by the same research (not part of Epic 4 but sequenced after it):

Phase Issue Slice Size Status
Later UH-74 Adversarial QA workflow profile (adversarial-qa) — OMX $ultraqa analog M shipped — #87 + #90
Later UH-75 Companion design.md artifact alongside mission.yaml — OMX $design analog S shipped — #88

Research source: docs/research/inspiration-systems.md §“oh-my-claudecode (OMC)”, §“oh-my-codex (OMX)”, §“Combined OMC/OMX takeaways”; verdicts in docs/research/adopt-reject-defer.md; side-by-side row in docs/research/comparison-matrix.md.

Discipline-layer slices from GSD-2 research

Section titled “Discipline-layer slices from GSD-2 research”

Standalone slices distilled from the GSD-2 / gsd-pi deep-dive (see docs/research/inspiration-systems.md §“GSD 2 (gsd-pi)”). They are orthogonal to Epic 3 and Epic 4 — each lands independently.

Phase Issue Slice Size Status
Core UH-76 Three-verdict runtime-result status (pass / needs-attention / needs-remediation) + uh mission verdict manual override — GSD-2 /gsd verdict analog S shipped — #88
Core UH-77 Drift detection + idempotent repair registry for uh validate — GSD-2 ADR-017 analog M shipped — #88
Core UH-78 uh status --json LLM-less query mode — GSD-2 gsd headless query analog; feeds Hermes Dashboard (UH-62) S shipped — #88
Later UH-79 Canonical docs/VISION.md with explicit “what we won’t accept” — GSD-2 VISION.md analog S shipped — #88
Later UH-80 Pre-inlined dispatch context contract — formalize the existing buildMissionPrompt shape across all four adapters M shipped — #88

Research source: docs/research/inspiration-systems.md §“GSD 2 (gsd-pi)”; verdicts in docs/research/adopt-reject-defer.md; side-by-side row in docs/research/comparison-matrix.md.

Build uh tui on OpenTUI — the same engine OpenCode uses in production (native Zig core, TypeScript bindings, Bun-first, MIT-licensed). Replaces today’s cat-driven mission review with a navigable live terminal app using OpenTUI’s Diff, Code (tree-sitter), ScrollBox, Select, and Input components.

Step Issue Slice Size Status
1 UH-45 Spike: install opentui, framework choice (vanilla / React / Solid), hello-world bound to UH state S shipped — Solid won; see docs/research/tui-framework.md
2 UH-46 Dashboard: live adapters + missions + sandboxes (three-pane) M shipped — Mission Control dashboard; see #57 + docs/architecture/tui.md
3 UH-47 Mission browser: drilldown with Code + Diff viewers M shipped — read-only mission artifact drilldown; see #59
4 UH-44 Mission run flow: trigger from TUI, stream events live M shipped — R opens a run dialog, live tail of events.ndjson, S stops via SIGTERM; see #63
5 UH-43 Adapter + sandbox manager: live checks, create/discard from inside M shipped — c re-check, n create-sandbox dialog, d discard with F force toggle; see #64
6 UH-42 Polish: keymap overlay, theming, error states, exit handling, Agent Skill install S shipped — ? overlay, per-project state persistence, operator runbook; see #61 + #65. Theming / $EDITOR / Ctrl+Z spun out as UH-48 / UH-49 / UH-50.

Shipped this cycle (2026-05-17 → 2026-05-19)

Section titled “Shipped this cycle (2026-05-17 → 2026-05-19)”

Adapter promoted to status: active on 2026-05-18 after live E2E smoke against hermes proxy start --provider nous. ToS-positioned subscription routing now first-class alongside hermes / codex / oh-my-pi.

Step Issue Slice PR
1 UH-36 Wire-format spike + recommendation #49
2 UH-35 Schema + manifest + template + dispatch stub #50
3 UH-39 Adapter impl: planner, runner, SSE parser, sentinel, blocked classification #51
4 UH-37 Live HTTP adapter check probe #53
5 UH-40 Architecture doc + setup runbook #54
6 UH-38 E2E smoke + promote to status: active #55
Issue Title PR
UH-45 TUI spike: opentui framework selection + hello-world #52
UH-46 TUI dashboard: live adapters + missions + sandboxes #57
UH-47 TUI mission browser: drilldown with Code + Diff viewers #59
UH-44 TUI mission run flow with live events.ndjson tail #63
UH-43 TUI adapter + sandbox manager (create/discard/recheck) #64
UH-42 TUI polish: keymap overlay (?) + per-project persistence + runbook #61 + #65

Runtime intelligence + operator polish + npm publish automation + Honcho memory (2026-05-19)

Section titled “Runtime intelligence + operator polish + npm publish automation + Honcho memory (2026-05-19)”
Issue Slice PR
UH-57 Runtime intelligence + operator polish: TUI screenshot capture, adapter-check age footer, runtime.cancelled event, mission capability enforcement #77
UH-58 Package renamed @agenticengineeringagency/ultimate-harness, publishConfig.access=public, CI dry-run + release publish workflow #79
UH-59 Honcho persistent-memory extension for the oh-my-pi adapter — env-driven enrich/record at the harness layer with fail-fast misconfig and graceful network degradation #83

Discipline layers — Spec-Driven, Test-Driven, Cross-runtime QA (shipped)

Section titled “Discipline layers — Spec-Driven, Test-Driven, Cross-runtime QA (shipped)”
Issue Slice PR
UH-54 SDD: structured acceptance_criteria with per-AC verify #68
UH-55 TDD: tdd workflow profile + test-first verify gate #69
UH-56 Cross-runtime QA: uh mission run-all + side-by-side comparison #70

All three layers compose: see docs/architecture/sdd-tdd-qa.md.

Shipped previous cycle (2026-05-13 → 2026-05-17)

Section titled “Shipped previous cycle (2026-05-13 → 2026-05-17)”
Issue Title
UH-23 Codex adapter — CLI transport via codex exec
UH-24 Cancelled (Anthropic ToS friction; superseded by UH-27 + UH-32)
UH-25 oh-my-pi adapter + runtime_config bucket + dispatch table + sandbox routing parity
UH-26 Per-runtime strict runtime_config validation (typo safety)
UH-27 Per-mission runtime_config_overrides + Anthropic-via-OMP path
UH-28 Runtime-final-message capture protocol (shared sentinel)
UH-29 createSandbox seeds bound mission into the worktree
UH-30 Codex CLI flag drift + child-stdin close + codex status: active
UH-31 Minimum Hermes ≥ 0.14.0 version pin
UH-33 runtime_config_overrides parity for hermes + codex
UH-34 Diff capture includes untracked new files

Adapter status as of 2026-05-29 (v0.9.0):

Adapter Status
hermes active (pinned ≥ 0.14.0)
codex active (verified against codex-cli 0.130.0 in UH-30 smoke)
hermes-proxy active (verified against hermes proxy start --provider nous in UH-38 smoke; manifest default nousresearch/hermes-4-405b)
openrouter active (OpenAI-compat HTTP to openrouter.ai; OPENROUTER_API_KEY; live-smoke verified)
pi active (vanilla pi agent CLI pi --print --mode json --no-session; shipped v0.7.0 #135/#150; live-smoke verified)
oh-my-pi active (graduated v0.8.0 #156; ToS posture per docs/runbooks/anthropic-via-omp.md §Auth & ToS posture; users opt in)
anthropic experimental (native pay-per-token Anthropic Messages API; ANTHROPIC_API_KEY env-only; shipped v0.9.0 #214; graduation pending live-smoke)

These are tracked in narrative form until they earn the priority to be filed:

Road to v1.0 is planned in docs/prds/uh-roadmap-to-1.0-v1.0-prd.md (milestones v0.8.0 → v1.0.0). Remaining proposals:

  • Native ANTHROPIC_API_KEY adapter — shipped in v0.9.0 ([#214], UH-136) as anthropic (status: experimental): the ToS-clean pay-per-token Messages-API path that complements UH-32. ANTHROPIC_API_KEY env-only, blocked classification, graceful no-key checker, optional live-smoke CI job. Graduation to active pending a live-smoke promotion record.
  • Container sandbox backend — shipped in v0.8.0 ([#155]/[#157]) behind the OpenSandbox seam, with #154 spike + ADR + runbook. Live docker-CLI fallback rejected as default by the ADR pivot rule.
  • Honcho-memory follow-ups — codex/hermes wired into enrichment + recordMissionExchange (v0.6.0); shipped in v0.9.0 ([#215], UH-137): honcho_search/honcho_remember harness-side memory operations + per-mission runtime_config.honcho_memory opt-out on every Honcho-aware adapter.
  • Telemetry instrumentation — an opt-in PostHog telemetry primitive shipped unwired in v0.9.0 ([#204]–[#206], UH-131); wiring the actual event emission is the UH-135 follow-up, which shipped with 0.11.0 (0.10.0 was skipped). Telemetry stays off unless opted in.
  • Capability-declaration enforcement — manifest/mission capabilities: binding (warn + --strict); shipped with 0.11.0 (0.10.0 was skipped). (Distinct from the runtime_requirements preflight, which always blocks.)
  • Incremental independent reviews — moved from docs/known-issues.md for 0.11.0-rc.1. uh mission review-prepare captures every file changed since the branch’s recorded base, and the reviewer must read each captured file in full, so a one-line fix forces a full re-review (a MiMo review of a 15-file slice took 504 s, and each follow-up fix invalidated it). Reviews and the uh land review gate could reuse an earlier review for files whose bytes did not change, the way uh acceptance rebind keeps evidence valid through input digests. Reviews are also slow for a second reason: a Command Code reviewer runs one tool call per turn and reads files over 40 KB in 600-line windows. Where the time goes per run has not been measured; uh report <run-id> reports model versus tool time.

Shipped (no longer proposals): the native anthropic pay-per-token adapter (v0.9.0 #214, experimental); Honcho memory operations + per-mission opt-out (v0.9.0 #215); cross-runtime QA uh mission run-all --runtimes with side-by-side diff/sentinel comparison; the Epic 2 TUI polish UH-48..53 (theme, $EDITOR, Ctrl+Z, tui screenshot, footer adapter-check age, runtime.cancelled on SIGTERM).

  • Muta integration — UH-as-consumer, UH-as-component, or independent? Needs a co-founder conversation. Deferred entirely to post-1.0 (see the road-to-1.0 PRD).

Resolved since: the Pi adapter shipped active in v0.7.0 (#135/#150); oh-my-pi graduated to active in v0.8.0 (#156); the native anthropic adapter shipped experimental in v0.9.0 (#214); Honcho memory operations + opt-out shipped in v0.9.0 (#215); mission capability-declaration enforcement shipped with 0.11.0 (manifest capabilities: binding, warn + --strict; 0.10.0 was skipped) and runtime_requirements preflight enforces compatibility.