Repository navigation
🐛🧑💻 - Expose completed progress assessment validation failures instead of opaque agent.failed #774
Description
Is there an existing issue for this?
- I have searched the existing issues.
Which actions are affected?
Commits
Which platforms are affected?
Linux
Description
A completed progress-agent result is rejected as assessment-invalid, but the workflow exposes only the generic error agent.failed with no offending requirement ID, field, anchor or schema reason. Please preserve the fail-closed outcome while distinguishing completed-result contract rejection from a transport/process failure, and publish the sanitized validation details needed to fix the result or its contract. A generic instruction to inspect the sanitized status is not actionable when the relevant validation reason is absent.
This report records one observed occurrence and one permitted same-head retry that concluded successfully. It does not assume a provider outage, invalid product scope, permission failure as the primary cause, or a deterministic regression. The diagnostic gap itself is directly observable. Existing progress must remain intact until a valid assessment exists.
Reproducing the issue
- Consumer: https://github.com/vypdev/geckocv/pull/39, issue Develop branch aligned with master #37 (private access required).
- Run https://github.com/vypdev/geckocv/actions/runs/40777808097, attempt 1. Workflow is Copilot - Commit.
- Exact base b461b78c035928856862b995d9008a64b3d59848; exact head ad9c0ff78cafe77552ff4589953f86133fc431c9.
- The workflow retrieves a bounded GitHub comparison, then reports two codex-cli 0.165.0 executions admitted and completed. Configured model is openai/gpt-6-luna. Logged prompt length is 90,229 characters; issue description length is 8,266.
- It rejects progress scope or requirement evidence as assessment-invalid and retains prior progress, but emits no rejected field/requirement/anchor detail.
- The terminal primary diagnostic is agent.failed, retryable Yes, reference 223de5e2-c094-44bf-b62b-44b3d38ad6a2.
- A failed-job-only retry was requested without source/scope/threshold changes. Attempt 2 concluded successfully on the same source head; the diagnostic gap on the rejected first attempt remains reproducible in its retained logs. The previous candidate passed this progress check; this candidate adds real boundary tests to meet the existing 100% Codecov patch requirement.
copilot Version
3.10.6; run setup resolves vypdev/copilot@v3 to f5e5a4c.
Relevant Log Output
Progress will be assessed from a bounded GitHub comparison pinned to the base and source heads.
Agent execution admitted (codex codex-cli 0.165.0).
Agent execution completed.
CheckProgress: prompt length=90229, issue description length=8266.
Agent execution admitted (codex codex-cli 0.165.0).
Agent execution completed.
Progress scope or requirement evidence was rejected (assessment-invalid). Existing progress was retained.
Impact: The admitted agent did not produce a usable result.
Error code: agent.failed
Action: Inspect the sanitized agent status and retry if appropriate.
Retryable: Yes
Reference: 223de5e2-c094-44bf-b62b-44b3d38ad6a2
Additional context and comments
The full approved scope is retained. Local verification has 41 passing architecture fixtures/policy tests, 100% line/branch checker coverage and unchanged thresholds; application CI remains separately authoritative. #773 tracks the distinct large-comparison recovery budget boundary.
A later optional native check publication also received 403 on checks.create via action-token-fallback. The installed consumer commit workflow declares contents: read and lacks the dedicated evidence credential route. Consumer issue https://github.com/vypdev/geckocv/issues/38 tracks supported setup/credential adoption. The assessment-invalid rejection occurred before that optional warning; please keep primary validation and secondary publication diagnostics distinct.
Suggested acceptance: retain invalid-result rejection and prior progress, expose bounded sanitized reason codes with the relevant requirement/field/anchor, distinguish transport/process failures from completed schema/scope/evidence rejection, state when a retry is useful, and test missing IDs, duplicate/out-of-scope requirements, invalid evidence references, malformed output and secondary check-publication failure. Do not accept invalid assessments, change consumer acceptance, bypass checks or reduce coherent work to recover publication.
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Activity
[-]Expose completed progress assessment validation failures instead of opaque agent.failed[/-][+]🐛 - Expose completed progress assessment validation failures instead of opaque agent.failed[/+]vypbot commentedon Oct 28, 2026
Hi! I’m @vypbot, the Copilot assistant for this repository.
I can answer questions, explain the codebase, propose implementation and test plans, review issues and pull requests for potential bugs or security problems, and help authorized maintainers apply changes.
Try
/copilot helpto see the available commands, or mention @vypbot with your question.efraespada commentedon Oct 28, 2026
Additional consumer reproduction at GeckoCV PR #40, exact head 3f67f9af7508161a45b2059e1817984a90b8537a: run https://github.com/vypdev/geckocv/actions/runs/40784130999, attempt 1, failed with the same assessment-invalid diagnostic and generic retryable agent.failed. Reference 99c50f61-9793-4780-bf06-f1c8a2c01f3b; prompt length 80690, issue description length 8492. Two Codex calls were admitted and completed; existing progress was retained. No field/path/requirement-specific rejection detail was provided. This is separate from the Linux Python setup failure in #776.
The consumer now uses the documented checks:write job credential as COPILOT_EVIDENCE_TOKEN on the trusted producer. The log confirms Copilot / Verification was published for this exact head; the earlier Checks 403 fallback warning did not recur. No PAT expansion or validation bypass was used. A source correction for the independent setup problem is being verified; no blind retry of this old head was requested.
efraespada commentedon Oct 28, 2026
Additional Domus reproduction: recurring completed-result rejection on two documentation-only heads
Latest update: The updated-head retry has now executed and failed with the same
assessment-invalidrejection. Its initial queue wait was normal and is not reported as a defect. Attempt 2 details and sanitized logs are included below.The Domus maintainer requested that Copilot defects be reported upstream rather than patched in the consumer. This is additional evidence for this existing issue, not a duplicate defect report.
Consumer and immutable scope
8ee7e1a15e91c91b6204ca97e8808c3207dee008(develop).vypdev/copilot@v3, resolved in the failed execution tof5e5a4cf9a80248994469cd88ad55eecab5b95ca; package version3.10.6.codex-cli 0.165.0; progress/findings provider/modelcodex / openai / gpt-6-luna.self-hosted, codex, Linux.bugbot-review-drafts: falsepolicy. That is separate from the failed progress assessment.Reproduction matrix
All times below are UTC; the maintainer's local reporting date is 2026-10-29.
fa7adf1118b85e1588f0de7d71a77f47822b5000ce2871b15508a1de53eaceca9bada001b31c25f2123119355063123165609347assessment-invalidat 21:50:14assessment-invalidat 23:37:52agent.failed, retryable Yesagent.failed, retryable Yes5292e49e-829c-49dc-be36-07f01fca9ac705275b28-f46f-4b27-93f9-898b14a0e432For the first head, attempt 2 completed successfully and published a successful Copilot / Verification check on the exact same SHA. The updated head's attempt 2 initially waited normally in the self-hosted queue, then executed and failed with the same validation rejection. It resolved the same Copilot source SHA, so this new occurrence is not attributed to a changed Action version.
Sanitized log evidence
First head:
Updated head:
What this establishes, and what it does not
assessment-invalid. The failed workflow log does not show the catalog validator causing this failure.Requested upstream investigation
Please retain strict invalid-result rejection and preservation of existing progress. Investigate why completed progress responses are repeatedly rejected, and surface bounded, sanitized validation reason/field/requirement/anchor coordinates so maintainers can act. Distinguish a completed response contract failure from agent transport/process failure and explain when retry can help.
A deterministic regression fixture for each discovered rejection cause should complement live-provider evidence; do not make an eventually successful model retry the only regression test. Do not publish raw rejected responses, reduce consumer scope, mark pending product work implemented, force a progress percentage, bypass checks, or weaken permissions/validation.
This report authorizes no Copilot implementation, deployment or lifecycle-label changes from the Domus contributor. The Copilot owners can triage and implement through their managed workflow.
Updated-head attempt 2 — completed failure, not pending capacity
ce2871b15508a1de53eaceca9bada001b31c25f2.vypdev/copilot@v3atf5e5a4cf9a80248994469cd88ad55eecab5b95caat 2026-10-28 23:57:00 UTC.codex-cli 0.165.0was admitted and completed for initial comparison handling, both evidence parts and the final assessment.assessment-invalidat 23:59:09 UTC.123176010208.agent.failed, retryable Yes; new reference4e4117ea-abaf-4fb1-86ed-12bb2633d692.The consumer PR remains honestly blocked by the failed current-head check. The prior head's successful retry is not evidence that this updated head passed. The separate greenfield catalog report is #777; its deterministic local validator result is not asserted as the progress-assessment root cause.
10 remaining items
vypbot commentedon Oct 29, 2026
Issue closed — historical progress
Current status: closed as completed.
The last estimate is retained below as history. Closure does not certify a new percentage, tests, or branch alignment.
Action required: none for this closed issue. If work continues, reopen the issue; a fresh assessment can update this same card.
Last assessment before closure (historical)
vypbot commentedon Oct 29, 2026
Issue closed: synchronization recommendation retired
No branch synchronization is required for this closed issue.
The previous recommendation is historical. Issue closure does not certify that the branches are synchronized.
efraespada commentedon Oct 29, 2026
Additional old-release observation, not a post-fix regression from GeckoCV's draft https://github.com/vypdev/geckocv/pull/42:
68710273d76fb101eac92218c17e825804a157ea.vypdev/copilot@v3tof5e5a4cf9a80248994469cd88ad55eecab5b95ca. The current v3 ref still has that SHA at inspection.assessment-invalid, prior progress retained, followed byagent.failed,Retryable: Yes, reference3e7318a8-fa66-46d3-9458-1e14f06e969a. The bounded validation reason is still absent in this old release.920c02e6cda8364c591284ca236e5662c83b3ae9on 2026-10-29 16:51:25Z; this run downloaded the older v3 at 16:51:43Z. It does not test that merged fix. No reopening or duplicate defect is requested from this old-release result.One bounded failed-job retry will be requested for this exact unchanged source under the reported retryable status. The failed attempt is retained; no scope, validation, branch ownership or coverage target is bypassed. Native acceptance of the diagnostic fix requires a run using a release that actually includes it.
Retry outcome: attempt 2 also failed as
assessment-invalidfollowed byagent.failed; reference9d887ffd-1bda-4d6a-8de4-0de3b354b951. Source remained68710273d76fb101eac92218c17e825804a157ea. No further retry is requested for this occurrence. This remains an old-release observation; the merged diagnostic fix has not been exercised by these attempts.Additional old-release diagnostic variant on the next source head
9553d61e5ef4cddafed9343cc88f119d52dc38b2: run https://github.com/vypdev/geckocv/actions/runs/40918977802 also resolves v3 tof5e5a4cf9a80248994469cd88ad55eecab5b95ca. The pinned comparison now requires three complete bounded parts. All three agent executions log completion; immediately after part 3 the log saysUnable to complete CheckProgressUseCase, and the final primary diagnostic is onlyworkflow.failed, retryable Yes, reference30051989-9334-4c46-92d9-a0065d59c5c1. No final-assessment diagnostic or underlying validation/aggregation reason is exposed. This is not evidence that the part output was valid, nor evidence that the primary cause was assessment-invalid.A later optional Checks publication separately receives 403 and explicitly states it does not determine workflow failure; it is not a demonstrated cause of the earlier CheckProgress failure. No retry is requested for this head. This also predates release adoption of #791 and does not establish a regression in that fix. Please include this completed-part/aggregation path when verifying bounded actionable diagnostics. Full product/milestone scope and actual coverage gates remain intact.
Latest old-release recurrence: source
d6e4d6ff9c0aec31b16e80f80c4623d4bfe4857e, run https://github.com/vypdev/geckocv/actions/runs/40923607167, resolves v3 tof5e5a4cf9a80248994469cd88ad55eecab5b95ca. Three bounded parts complete; final progress is rejected asassessment-invalid, then onlyagent.failed, retryable Yes, referencef27b8428-4cec-41bd-851f-da64a01ac032. Prior progress is retained; no offending result field/requirement/anchor is exposed. No retry requested on this head, no model-output validity assumed, and no post-fix regression claimed. Local backend gates now pass 1,122 tests with original floors and unchanged canonical wire shapes; full milestone acceptance and the 100% patch target remain explicitly pending.Further old-release recurrence on the canonical compatibility checkpoint: source
1997175bcd473754cfcd675f1deda1189afc456a, run https://github.com/vypdev/geckocv/actions/runs/40948795270, attempt 1. The primary job setup explicitly resolvesvypdev/copilot@v3tof5e5a4cf9a80248994469cd88ad55eecab5b95ca(3.10.6); this does not exercise #791. All seven bounded evidence-part agents and the final assessment agent log completion using codex-cli 0.166.0. Final progress is rejected asassessment-invalid, followed byagent.failed, retryable Yes, referenceeba7eb9a-fd44-4e7c-b6eb-e5221531d446. Prior progress remains intact; no rejected requirement/field/anchor reason is exposed. No retry, reopening or post-fix regression claim is requested.The preceding
5cac606source passed backend/frontend CI and Copilot Commit/Branch Sync/PR workflows. The new source frontend CI passes; backend verification is still in progress at inspection. Local scoped verification passes 57 methods covering real compiler/serializer consumers, directional policy and disposable Git baseline histories, with all nine Python production modules at 100% measured lines/branches and unchanged floors. The independent 275,953-byte frozen reference retains all 61 operations/187 variants/74 components and explicitly remains an initial human-review proposal. Complete milestone acceptance and the 100% patch target remain open. These execution results do not establish that the rejected model assessment was valid; full scope and strict validation are retained.Additional old-release completed-part diagnostic gap at source
3ab66e49376971ca7cc08c52e0a07ec5807f8258, run https://github.com/vypdev/geckocv/actions/runs/40959930477, attempt 1. Setup explicitly resolvesvypdev/copilot@v3tof5e5a4cf9a80248994469cd88ad55eecab5b95ca; this does not exercise #791. Eight complete parts are planned. The initial comparison agent and part 1/part 2 agents log completion with codex-cli 0.166.0. Immediately after part 2, CheckProgressUseCase fails, before parts 3-8 or final assessment, with onlyworkflow.failed, retryable Yes, reference2e26a965-1cc4-4cc4-a1c6-19c12d654f07. No underlying field/partition/aggregation reason is exposed. Existing persisted state is retained. Neither result validity nor assessment-invalid is inferred; a later optional Checks 403 is secondary. No retry, reopening or post-fix regression is requested.The consumer retains full scope and acceptance. This candidate pins a released JaCoCo analyzer after a same-data experiment and real Kotlin RED/GREEN fixture with genuinely missed authored-code controls. All ten coverage-policy fixtures and 1,144 backend tests pass locally with unchanged floor/exclusion policy; new effective denominators are explicitly distinguished from new behavior. Prior source 8981454 backend/frontend and Copilot checks pass, but its 93.11% patch summary still fails the unchanged 100% target. Current candidate application CI and complete milestone acceptance remain separate.
Additional old-release completed-part diagnostic gap at source
9d736ed1cb7d9dc808f12aaa55c011c98d907969, run https://github.com/vypdev/geckocv/actions/runs/40965767729, attempt 1. The setup log explicitly downloadsvypdev/copilot@v3atf5e5a4cf9a80248994469cd88ad55eecab5b95ca(3.10.6); this does not exercise #791. Nine complete comparison parts are planned against base5dde8907973d3e9718c9e5886daa8a6caf7ae078. The initial agent and parts 1-4 log completion with codex-cli 0.166.0; immediately after part 4, CheckProgressUseCase fails before parts 5-9/final assessment. The primary diagnostic is onlyworkflow.failed, retryable Yes, reference7f09339c-a3c2-4e12-8f2c-a49a730c921f. No validation/partition/aggregation cause is exposed. A later optional Checks 403 explicitly does not determine workflow failure. Neither result validity nor assessment-invalid is inferred. No retry, reopening or post-fix regression is requested.The consumer corrects a reproduced path/query resource-identity collision and adds 31 scoped HTTP cases; full local backend gates pass with 1,175 tests and unchanged floors/exclusions. Nineteen compiled reference/exporter fixtures, 57 typed tooling methods and 44 separate overlapping guards pass. The fresh 61-operation/187-variant/74-component export is byte-identical to the independently frozen proposal. Current source backend/frontend CI and full patch acceptance remain separate. Full product scope, strict assessment validation and human baseline/adoption gates are preserved.
Latest old-release completed-part observation at source
72f0c02f9b756ace9bb7f7b5d71c8e28684a6473, run https://github.com/vypdev/geckocv/actions/runs/40969960179, attempt 1. Setup resolvesvypdev/copilot@v3tof5e5a4cf9a80248994469cd88ad55eecab5b95ca(3.10.6). Nine complete parts are planned; the initial comparison agent and parts 1-4 complete, then CheckProgressUseCase exposes onlyworkflow.failedbefore parts 5-9/final, retryable Yes, reference29cb3cc0-c280-4e71-9c60-1bb186865de2. Later optional Checks 403 is secondary. No result-validity, assessment-invalid, retry/reopening or post-fix regression is inferred.The consumer now has 1,188 passing local backend cases with original floors/exclusions, explicit required-path readers and synthetic-credential redaction. Real HTTP REDs precede the fixes; all 1,297 measured changed Kotlin source-line coordinates have covered instructions/no missed branches in aggregate. That is not every-instruction, whole-project or remote Codecov acceptance. Fresh wire bytes remain identical to the independently frozen proposal; initial human baseline adoption stays pending. Nineteen reference/exporter methods, 57 typed methods and 44 separately overlapping guards pass. Current application CI remains separate; full approved product scope is retained.
Additional old-release observation, not a post-fix regression: GeckoCV source
2379ca938f8a93a8c8d2d767c38726b5d078fdec, run https://github.com/vypdev/geckocv/actions/runs/40974855229, attempt 1. Setup explicitly downloads v3 atf5e5a4cf9a80248994469cd88ad55eecab5b95ca/ 3.10.6. Nine complete parts are planned; the initial agent and parts 1–3 log completion, then CheckProgressUseCase exposes onlyworkflow.failedbefore parts 4–9/final, referencea68953df-9550-4f7d-a423-05a759a20f8e. Optional Checks 403 remains secondary. No result validity, assessment-invalid cause, retry/reopening or regression of the merged fix is inferred.The consumer retains all product scope, evidence and thresholds. Three administrative browser preservation cases pass before/after a mounted-node/function-binding refactor; a fourth covers late read completion after unmount. Full local frontend has 296 passing cases and 85 sequential E2E without failures/skips/flakes, and current frontend CI passes. Backend sources retain the 1,188-case receipt at 72f0c02; the new backend CI is still running at observation. Two changed components have full measured local coverage, not a whole-project or final remote acceptance claim. No generated-code edit, coverage exclusion or scope reduction is used.
efraespada commentedon Oct 30, 2026
Additional retained observation on the same published old runtime, not a regression claim against this issue's develop fix:
No unchanged-state rerun, source/acceptance reduction, release or upstream branch was requested. Existing source verification and full product scope are preserved. This occurrence supplies another exact run for validating the already owned diagnostic fix when it is published; its internal rejection cause is not inferred from the generic result.