feat(e2e): run long-horizon Bub workloads - #1232
Open
PsiACE wants to merge 2 commits into
Open
Conversation
This was referenced Aug 13, 2026
PsiACE
marked this pull request as ready for review
August 13, 2026 11:10
Contributor
There was a problem hiding this comment.
Pull request overview
Implements the “long-horizon Bub runtime” layer for the e2e harness by replacing the prior scenario-replay flow with a single workload pipeline that runs tasks through Harbor → ACP → Bub, writes normalized evidence, and evaluates Memory behavior via a common acceptance contract.
Changes:
- Reworks the e2e runner and CLI to execute catalog-defined workloads via Harbor/ACP/Bub and evaluate them with a normalized evidence + Memory evaluation/reporting stack.
- Updates the fixed Compose harness (DinD entrypoint, mounts, env wiring) and Make targets/CI workflow to run
acceptancewith repeatable--id/--categoryselectors. - Removes the older scenario YAMLs, Bub span tracing plugin, and run.sh unit tests in favor of the new runtime/evidence model.
Reviewed changes
Copilot reviewed 27 out of 28 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| Makefile | Consolidates harness entrypoints around acceptance and routes selectors via $(ARGS). |
| e2e/bub/tests/test_run_script.py | Removes prior unit tests that validated run.sh cleanup/exit-code behavior. |
| e2e/bub/tests/test_long_horizon_acceptance.py | Adds behavior coverage for Memory acceptance semantics independent of Harbor pass/fail. |
| e2e/bub/tests/test_evidence_redaction.py | Updates redaction/evidence tests to the new observation/evaluation schema and sinks. |
| e2e/bub/src/powercontext_e2e/tracing.py | Removes Bub OpenTelemetry span plugin that was specific to the old replay approach. |
| e2e/bub/src/powercontext_e2e/settings.py | Adds typed harness-owned settings (repo mount, secrets redaction set, commit inference). |
| e2e/bub/src/powercontext_e2e/runner.py | Replaces scenario replay runner with Harbor Job execution + evidence capture + probe collection. |
| e2e/bub/src/powercontext_e2e/rescore.py | Adds offline rescoring for recorded evidence without re-executing the adapter. |
| e2e/bub/src/powercontext_e2e/report.py | Adds Marko-based human-readable report rendering for the new evaluation report model. |
| e2e/bub/src/powercontext_e2e/models.py | Introduces normalized evidence/evaluation models (TaskObservation, EvaluationReport, artifacts, capture records). |
| e2e/bub/src/powercontext_e2e/harbor_agent.py | Adds a Harbor ACP agent implementation that installs Bub + ACP server in the agent env. |
| e2e/bub/src/powercontext_e2e/evidence.py | Adds shared evidence utilities (redaction, instruction loading, artifact fingerprinting). |
| e2e/bub/src/powercontext_e2e/evaluation.py | Adds Memory evaluation logic over normalized evidence (coverage, groundedness, probe support, thresholds). |
| e2e/bub/src/powercontext_e2e/catalog.py | Extends catalog contracts (native artifact set) and selection logic used by acceptance. |
| e2e/bub/src/powercontext_e2e/artifacts.py | Adds final artifact sinks (replay/eval JSON + rendered Markdown), with redaction. |
| e2e/bub/src/powercontext_e2e/main.py | Updates CLI to acceptance selectors + rescore, and centralizes HarnessSettings construction. |
| e2e/bub/scenarios/project-database-decision.yaml | Removes obsolete scenario YAML in favor of task manifests. |
| e2e/bub/scenarios/locomo-support-group.yaml | Removes obsolete scenario YAML in favor of task manifests. |
| e2e/bub/run.sh | Refactors Compose harness script to a command-based interface and forwards workload selectors to acceptance. |
| e2e/bub/README.md | Updates documentation to the workload catalog architecture, unified execution path, and selection semantics. |
| e2e/bub/pyproject.toml | Updates dependencies to support Harbor, Marko reporting, and settings management; removes Bub entry-point tracing plugin. |
| e2e/bub/harbor-task-overlay.yaml | Adds overlay compose tweaks for Harbor-executed tasks (host-gateway + label disable). |
| e2e/bub/Dockerfile | Extends harness container to support nested Docker (DinD) and uses a container entrypoint wrapper. |
| e2e/bub/container-entrypoint.sh | Adds DinD bootstrap + socat port forwarding and then execs the CLI inside the container. |
| e2e/bub/compose.yaml | Updates env wiring/timeouts, enables privileged DinD, mounts Codex auth + Docker data volume. |
| docker/Dockerfile | Switches uv base image reference to docker.io mirror. |
| .github/workflows/e2e-harness.yml | Updates cleanup to use make harness-compose-down after matrix runs. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+15
to
+18
| def fingerprint(path: Path, *, relative_to: Path | None = None) -> NativeArtifact: | ||
| content = path.read_bytes() | ||
| name = path.relative_to(relative_to).as_posix() if relative_to is not None else path.name | ||
| return NativeArtifact(name=name, sha256=sha256(content).hexdigest(), bytes=len(content)) |
Comment on lines
+293
to
+301
| def _load_capture_records(trial_dir: Path) -> tuple[CaptureRecord, ...]: | ||
| records: list[CaptureRecord] = [] | ||
| for path in sorted(trial_dir.rglob("powercontext-capture.jsonl")): | ||
| records.extend( | ||
| CaptureRecord.model_validate_json(line) | ||
| for line in path.read_text(encoding="utf-8").splitlines() | ||
| if line.strip() | ||
| ) | ||
| return tuple(records) |
Comment on lines
+7
to
+10
| command=${1:-acceptance} | ||
| if [ "$#" -gt 0 ]; then | ||
| shift | ||
| fi |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 27 out of 28 changed files in this pull request and generated no new comments.
Suppressed comments (3)
e2e/bub/run.sh:10
- e2e/bub/run.sh now has new argument parsing/dispatch logic (command shifting and forwarding remaining args), but the previous dedicated run.sh tests were removed and there is no remaining pytest coverage for the shell entrypoint. This makes regressions in cleanup, argument validation, and selector forwarding harder to catch.
command=${1:-acceptance}
if [ "$#" -gt 0 ]; then
shift
fi
e2e/bub/src/powercontext_e2e/runner.py:297
- _load_capture_records reads each powercontext-capture.jsonl with read_text().splitlines(), which loads the entire capture log into memory. Long-horizon workloads can produce large capture logs, so this can cause unnecessary memory spikes in the harness process.
def _load_capture_records(trial_dir: Path) -> tuple[CaptureRecord, ...]:
records: list[CaptureRecord] = []
for path in sorted(trial_dir.rglob("powercontext-capture.jsonl")):
records.extend(
CaptureRecord.model_validate_json(line)
e2e/bub/src/powercontext_e2e/evaluation.py:96
- The metrics field "captured_sources" currently counts captured records (events), not unique sources. This is misleading for diagnostics and makes it harder to interpret capture coverage/groundedness alongside the metric name.
metrics = {
"capture_events": len(eligible_records),
"captured_sources": len(captured_records),
"completed_checkpoints": len(completed_checkpoints),
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack
Merge in this order. This PR depends on #1231.
Which issue or RFC does this PR close?
Implements the runtime layer of #1229.
Rationale for this change
Acceptance should have one execution path while keeping deterministic database scenarios distinct from Bub-backed workloads.
What changes are included in this PR?
acceptancecommand with repeatable--idand--categoryfilters.rescore.model: falseworkloads in default CI and keep Terminal-Bench opt-in.Are there any user-facing changes?
The e2e harness uses
acceptanceas its single execution command. SQLite and OceanBase remain deterministic acceptance workloads, not Bub workload types.How was this change tested?
make checkmake testmake docs-testmake contract-testmake harness-checkmake harness-compose-checksh -n e2e/bub/run.shThe model-backed Terminal-Bench acceptance was not run locally.
AI usage statement
OpenAI Codex (GPT-5) assisted with implementation, tests, and documentation. The author is responsible for the submitted changes.