[ronniegeraghty]

Summary

Remove the project Docker-backend default that the GitHub Copilot Evals tools do not support yet. With no override, native Vally 0.21.0 selects its local host backend.

• Remove defaults.backend and its Docker-specific grade-network argument from .vally.yaml.
• Remove mandatory Docker/Compose preparation from the manually dispatched eval workflow; restore its repository-pinned host Copilot launcher.
• Remove the obsolete Docker prerequisite from the README Requirements checklist and update execution/authentication/model guidance.
• Replace Docker-default regressions with native local-default and supported-project-root checks. Preserve credential restoration/failure/empty-input and independent-reviewer coverage.

This is separate from #15. Dependency pins/lock, standard models, plugin sources, evals, behavior graders, fixtures and historical saved evidence are unchanged. No unrelated complexity proposals or plugin upgrades are included.

Compatibility and exposure

The marketplace pins Evals 0.24.0 at 34ddb5803aeaee8330ab70c19ad1a3f37fb77ded. Its full-engine reader, and the separately checked locally installed 0.24.6 reader, accept project roots paths, suites, environments, params, metadata, but reject defaults. Both exact sources were inspected without installing or executing the plugin. The regression guard covers their shared root-key shape, not an installed-version or end-to-end execution test.

This intentionally restores local execution, not a Docker-failure fallback. Tasks, setup and graders run on the host. Workspaces and narrow credentials are not filesystem, credential, network or shared-kernel isolation. Existing run/permission/cost approval requirements remain; no authentication or organization/repository settings change.

Validation and review fixes

• Initial 31 focused model-free checks and lint passed; the affected backend/documentation checks and lint were rerun for the review fixes.
• Native standard-model --dry-run produces 14 plans across seven retained evals and two unchanged models.
• Native project resolution selects local with no backend arguments for every retained eval; dependency versions and CLI aliases are unchanged.
• Review 1's obsolete Requirements bullet was removed in 5119b6e. Review 2's version-evidence distinction was addressed by verifying the exact pinned reader and documenting the boundary in 451238c.
• Review 3's historical-reference concern was addressed in 868d83b: the guide links the original Docker run summary in #17 and explicitly distinguishes its reported outcomes/local saved paths from a published, independently inspectable raw-results artifact. The historical broad-credential record does not validate narrow-token authentication or this local-default update; saved files remain untouched.
• No paid eval, authentication execution, Docker operation or host dependency install was performed. These checks do not claim live paid-tool execution or behavioral success.

Current head: 868d83b3edc811556c26deca7261802cbdabad7a. A fresh Copilot review was actually requested for this head and confirmed by GitHub review_requested event 35986320665 at 2026-11-13T02:39:47Z. This confirms the request, not completion or sign-off. Current-head CI and the completed review will be collected before handing back; no merge is authorized.

[Copilot]

🟡 Changes recommended

The README still requires Docker for paid repository evaluations despite this change removing that dependency, leaving the setup instructions inconsistent.

1 open issue

• README still requires Docker for paid repository evals · New

What changed in this PR

Removes the unsupported project-level Docker default from the shared eval configuration, allowing native Vally to select local host execution.

Changes:

• Removes Docker backend settings and workflow prerequisites, restoring the repository-pinned host Copilot launcher.
• Updates execution, authentication and model guidance to explain host execution and exposure limits.
• Replaces Docker-specific regressions with local-default and supported-project-field checks while preserving credential coverage.

File	Description
README.md	Updates paid-eval guidance for local execution.
docs/​checks-and-evals.md	Documents compatibility, host execution and scoped authentication.
.vally.yaml	Removes Docker backend and grading-network defaults.
.github/​workflows/​evals.yml	Restores the host Copilot launcher and removes Docker readiness checks.
.github/​scripts/​backend-defaults.test.mjs	Checks local backend inheritance and supported configuration fields.

_{Review effort: Max (dynamic)}

[ronniegeraghty]

The single README prerequisite finding from review 1 is addressed in 5119b6e. The focused correction is pushed; CI and a completed review on the changed head remain to be collected.

_{🤖 AI-authored reply via Agency Finish PR.}

[Copilot]

🔵 Needs a closer look

The compatibility documentation does not distinguish the checked Evals 0.24.6 from the marketplace-pinned 0.24.0, potentially implying validation of the installed version.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Clarify compatibility check targets Evals 0.24.6, not pinned 0.24.0

docs/​checks-and-evals.md:134

marketplace.json:31–34 installs Evals 0.24.0, but this compatibility check and the allowlist in .github/scripts/backend-defaults.test.mjs:34–35 target 0.24.6. Make that distinction explicit, or verify the pinned reader, so users do not treat this check as validation of the version installed from this marketplace.

1 resolved since last review

• README still requires Docker for paid repository evals

_{Review effort: Max (dynamic)}

[ronniegeraghty]

Verified the exact marketplace-pinned Evals 0.24.0 reader at catalog commit 34ddb580 as well as local 0.24.6. Both full-engine sources use the same project root-key allowlist. 451238c now names both versions explicitly and limits the regression claim to source/configuration shape, not an installed-version or live execution test. No plugin upgrade was made.

_{🤖 AI-authored reply via Agency Finish PR.}

[Copilot]

🔵 Needs a closer look

Human judgment is needed to assess the significance of unlinked references to prior Docker validation and saved results that readers cannot inspect.

No open issues 🎉

_{Review effort: Max (dynamic)}

[ronniegeraghty]

Addressed the historical-evidence reference concern in 868d83b3edc811556c26deca7261802cbdabad7a. The guide now links the original PR17 run summary and explicitly distinguishes its reported outcomes/local saved paths from a published, independently inspectable raw-results artifact. That historical broad-credential record is not proof of narrow-token authentication or the restored local backend; the saved files remain untouched.

The five existing backend/credential-documentation checks and lint pass. No model run, credential execution, raw-result publication, dependency/settings change or Docker operation was performed. Current-head CI and review outcomes are not presumed from the previous head.

_{🤖 AI-authored reply via Agency Finish PR.}

[Copilot]

🔵 Needs a closer look

Moving eval setup, tasks, and graders from Docker to developer and CI hosts removes container isolation and offline grading enforcement, requiring human acceptance of the changed filesystem, network, and credential-exposure risks.

No open issues 🎉

_{Review effort: Max (dynamic)}

[ronniegeraghty]

The current-head review's exposure warning is correct and is the intended compatibility tradeoff, not an isolation-equivalent replacement. Removing the unsupported Docker default also removes its container boundary and grade-network: none enforcement. Setup, tasks, tools and graders run on the developer/CI host; filesystem, network and model-credential exposure remain explicitly documented. Narrow credentials and disposable workspaces are not a sandbox, and each actual run still needs its normal permissions/cost approval.

No additional code change or same-head review request is being made to conceal that risk. Maintainer acceptance remains human judgment; I am not treating “No open issues” or green CI as security sign-off. The reviewed head is 868d83b3edc811556c26deca7261802cbdabad7a; no paid eval or merge was performed.

_{🤖 AI-authored reply via Agency Finish PR.}