Add regression coverage for Z.AI input and cache token totals by Copilot · Pull Request #75383 · github/gh-aw

Copilot · 2026-11-06T08:53:48Z

Z.AI’s Anthropic-compatible endpoint reported zero input and cache-read tokens in proxy summaries despite nonzero CLI counts. The current AWF pin already includes the upstream capture fix; this PR adds downstream regression coverage.

CLI aggregation: Assert input, output, cache-read, and per-model totals.
Detection reporting: Assert normalized counts reach the summary and usage artifact unchanged.
Scope: Test-only changes using a synthetic record with the reported totals: 38,719 input, 4,058 output, and 112,640 cache-read tokens.

Fixes API proxy token usage reports 0 input and cache tokens for an Anthropic-compatible Z.AI endpoint #75271

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

github-actions · 2026-11-06T13:56:26Z

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · routed: gpt56 max · ~1 AIC · ⌖ ~2 AIC · ⊞ 7.7K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

github-actions · 2026-11-06T13:56:26Z

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

github-actions · 2026-11-06T13:56:38Z

✅ PR Specification Reviewer completed successfully!

PR Specification Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from PR Specification Reviewer · gpt56 · ~2 AIC · ⌖ ~1 AIC · ⊞ 1.6K · ◷

Generated by PR Specification Reviewer for #75383

github-actions · 2026-11-06T13:56:39Z

✅ Ponytail Reviewer completed successfully!

Generated by Ponytail Reviewer for #75383

github-actions · 2026-11-06T13:56:42Z

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

github-actions · 2026-11-06T13:56:44Z

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · routed: gpt56 max · ~1 AIC · ⌖ ~1 AIC · ⊞ 6.3K · ◷

🧪 Test quality analysis by Test Quality Sentinel

github-actions · 2026-11-06T14:03:04Z

Comment Memory

Peek at saved memory (pr-code-quality-reviewer)

reviewed_at: 2026-11-06T13:59:01Z
review_event: COMMENT
top_themes: [locale-dependent test formatting]
files_reviewed: [actions/setup/js/parse_token_usage.test.cjs, pkg/cli/token_usage_test.go]
comment_count: 1

Note

This comment is managed by comment memory.

Expand the saved memory block to view or edit the persistent context for this thread.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~1 AIC · ⊞ 1.6K · ◷
_{Comment /review to run again}

github-actions

Reviewed the test-only changes. The added coverage is useful and the aggregation assertions are aligned with the reported regression.

Highlights

One portability concern: the JavaScript summary assertion hard-codes US comma formatting despite locale-sensitive rendering; see the inline comment.
No correctness or security-blocking issues found in the changed lines.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~1 AIC · ⊞ 1.6K
_{Comment /review to run again}

github-actions · 2026-11-06T14:03:07Z

+        cache_write_tokens: 0,
+      });
+      const markdown = mockCore.summary.addRaw.mock.calls[0][0];
+      expect(markdown).toMatch(/\|\s*38,719\s*\|\s*4,058\s*\|\s*112,640\s*\|\s*0\s*\|/);


This assertion makes the regression test locale-dependent, so CI on a non-US locale can fail even when token aggregation is correct.

💡 Make the expected formatting deterministic

toLocaleString() uses the runner locale, while this regex hard-codes comma grouping (38,719). Pass an explicit locale in the production/test formatter or assert the parsed table row using a locale-independent representation; also scope the match to the intended model row so another row cannot satisfy it accidentally.

Updated the row check to use the runtime locale’s number formatting and scope the match to the glm53 row. Commit: 36744f1.

Copilot

🟢 Approval recommended

The supplied status records no active issues for this regression-coverage change, which targets preservation of Z.AI token totals through aggregation and detection reporting.

No open issues 🎉

What changed in this PR

Adds regression coverage for preserving Z.AI input, output, cache-read, and cache-write token totals through CLI aggregation and detection reporting.

Changes:

Adds Go tests for aggregate and per-model token totals.
Adds JavaScript tests for detection summaries and usage artifacts.
Covers the reported totals: 38,719 input, 4,058 output, and 112,640 cache-read tokens.

File	Summary	Findings
`pkg/cli/token_usage_test.go`	Verifies aggregate and per-model Z.AI token totals.	No issues identified.
`actions/setup/js/parse_token_usage.test.cjs`	Verifies detection summary rendering and usage artifact preservation.	Two moderate issues: the test does not configure and verify the detection artifact path, and its summary assertion is not anchored to the aggregate total row.

This pull request requires changes.

_{Review effort: Lite (auto)}

github-actions

What this change does:

Adds regression coverage for normalized Z.AI Anthropic-compatible usage in the Go parser and JavaScript detection path. It checks aggregate and per-model totals, summary rendering, and artifact output.

Nice to have

Derived total assertion adds no coverage (pkg/cli/token_usage_test.go:L63)
- What this is: This assertion recomputes a value from the three token totals already asserted above.
- Problem: It cannot catch a production regression independently; when those assertions pass, the sum is necessarily 155417, so it only adds another literal to maintain.
- Fix: Delete the assertion.
- If we skip it: The test carries a redundant check and a second place to update when the fixture totals change.

Verdict: Ship.
Lean: -1 lines possible.

Generated by ✂️ Ponytail Reviewer for #75383 · codex · routed: gpt56 max · ~4 AIC · ⌖ ~1 AIC · ⊞ 118
_{Comment /ponytail to run again}

github-actions · 2026-11-06T14:03:24Z

+		assert.Equal(t, 4058, summary.TotalOutputTokens)
+		assert.Equal(t, 112640, summary.TotalCacheReadTokens)
+		assert.Zero(t, summary.TotalCacheWriteTokens)
+		assert.Equal(t, 155417, summary.TotalInputTokens+summary.TotalOutputTokens+summary.TotalCacheReadTokens)


What this is: This assertion recomputes a value from the three token totals already asserted above. Problem: It cannot catch a production regression independently; when those assertions pass, the sum is necessarily 155417, so it only adds another literal to maintain. Fix: Delete this assertion. If we skip it: The test carries a redundant check and a second place to update when the fixture totals change.

Removed the redundant derived-total assertion. Commit: 36744f1.

github-actions

Skills-Based Review 🧠

Applied /tdd to the two new regression tests. No blocking issues. Two minor test-quality suggestions are posted inline.

📋 Key Themes & Highlights

Key Themes

Test style: the JS test reads addRaw.mock.calls[0][0] directly, while the rest of the file uses toHaveBeenCalledWith(expect.stringContaining(...), true).
Redundant assertion: the Go sum check (155417) repeats the three field assertions above it.

Positive Highlights

✅ Both tests use the same synthetic record, so the Go and JS coverage stay aligned.
✅ The JS test checks that the source JSONL is unchanged, which guards the artifact-preservation claim.
✅ The Go subtest passes locally (go test -run TestParseTokenUsageFile, subtest Z.AI_Anthropic-compatible_usage).

Verification gap

I could not run the JS test. npm ci fails in this sandbox on a TLS certificate error from the npm registry, and vitest is not installed. I checked the JS assertions by reading the source: the table regex matches the column order in generateTokenUsageSummary, the addRaw mock is reset in beforeEach, and the file paths honour GH_AW_TMP_DIR. Please confirm the JS test in CI.

@copilot please address the review comments above.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~4 AIC · ⌖ ~1 AIC · ⊞ 5.4K
_{Comment /matt to run again}

github-actions · 2026-11-06T14:08:41Z

+        cache_read_tokens: 112640,
+        cache_write_tokens: 0,
+      });
+      const markdown = mockCore.summary.addRaw.mock.calls[0][0];


[/tdd] The row check reads mock.calls[0][0], which depends on no earlier addRaw call and couples the test to call order. Use the toHaveBeenCalledWith(expect.stringContaining(...), true) style that other tests in this file already use, so the assertion reads as a spec.

💡 Suggested change

expect(mockCore.summary.addRaw).toHaveBeenCalledWith(expect.stringMatching(/\|\s*38,719\s*\|\s*4,058\s*\|\s*112,640\s*\|\s*0\s*\|/), true);

This also removes the local markdown variable.

@copilot please address this.

Changed the assertion to toHaveBeenCalledWith and made it target the glm53 row using locale-aware values. Commit: 36744f1.

github-actions · 2026-11-06T14:08:41Z

+		assert.Equal(t, 4058, summary.TotalOutputTokens)
+		assert.Equal(t, 112640, summary.TotalCacheReadTokens)
+		assert.Zero(t, summary.TotalCacheWriteTokens)
+		assert.Equal(t, 155417, summary.TotalInputTokens+summary.TotalOutputTokens+summary.TotalCacheReadTokens)


[/tdd] This sum check repeats the three field assertions above it and hard-codes a derived total (155417). It adds no coverage and goes stale if the fixture changes. Consider removing it.

@copilot please address this.

Removed the redundant derived-total assertion. Commit: 36744f1.

gh-aw-bot · 2026-11-06T14:26:53Z

@copilot

Copilot assignment requested for the following outstanding work:

Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
Review (actions/setup/js/parse_token_usage.test.cjs:454): This assertion makes the regression test locale-dependent, so CI on a non-US locale can fail even when token aggregation is correct. - Add regression coverage for Z.AI input and cache token totals #75383 (comment)
Review (pkg/cli/token_usage_test.go:63): What this is: This assertion recomputes a value from the three token totals already asserted above. Problem: It cannot catch a production regression independently; when those assertions pass, the sum is necessarily 155417, so it only adds another literal to maintain. Fix: Delete this assertion. If we skip it: The test carries a redundant check and a second place to update when the fixture totals change. - Add regression coverage for Z.AI input and cache token totals #75383 (comment)
Review (actions/setup/js/parse_token_usage.test.cjs:453): [/tdd] The row check reads mock.calls[0][0], which depends on no earlier addRaw call and couples the test to call order. Use the toHaveBeenCalledWith(expect.stringContaining(...), true) style that other tests in this file already use, so the assertion reads as a spec. - Add regression coverage for Z.AI input and cache token totals #75383 (comment)
Review (pkg/cli/token_usage_test.go:63): [/tdd] This sum check repeats the three field assertions above it and hard-codes a derived total (155417). It adds no coverage and goes stale if the fixture changes. Consider removing it. - Add regression coverage for Z.AI input and cache token totals #75383 (comment)

Once you start work, post this work summary as a PR comment without the leading Copilot mention, including the Sous-chef head, work, and state lines below, to record that the assignment is running. Apply the necessary fixes on this PR's existing branch, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved; stop when only human review or CI remains. Do not merge or trigger CI.

Sous-chef requested-head: 2d2968b
Sous-chef requested-work: 3016fe302d5fae87f0ba5bef4f68ef0b30efbf907567646679b7096006b30edc b7d6859d6c602a858c4d8402900feb83d6686afd764099f10193de298d92007d bbe19b039bf872c4fdd50cd9fa36da096feaf07cbdaef1506549eee97d5383b7 f7a47e6b9e85b95de7cf30691eec12cd27fe24bec34e2c7d13f73422c2ac7743
Sous-chef requested-state: 38fb0b0b207d11f81048e6ad36c8eebcbf69dbba9929b1cf8d861241e2018fa1

Generated by 👨🍳 PR Sous Chef · pi · haiku45 · ~9 AIC · ⌖ ~11 AIC · ⊞ 1.2K · ◷
_{Comment /souschef to run again}

Copilot · 2026-11-06T14:32:04Z

@copilot

Copilot assignment requested for the following outstanding work:...

Sous-chef requested-head: 2d2968b
Sous-chef requested-work: 3016fe302d5fae87f0ba5bef4f68ef0b30efbf907567646679b7096006b30edc b7d6859d6c602a858c4d8402900feb83d6686afd764099f10193de298d92007d bbe19b039bf872c4fdd50cd9fa36da096feaf07cbdaef1506549eee97d5383b7 f7a47e6b9e85b95de7cf30691eec12cd27fe24bec34e2c7d13f73422c2ac7743
Sous-chef requested-state: 38fb0b0b207d11f81048e6ad36c8eebcbf69dbba9929b1cf8d861241e2018fa1

Work is underway: merge the latest main, fix the locale-sensitive and call-order-dependent JavaScript assertion, remove the redundant Go total assertion, run focused validation, then report and resolve the review threads.

Initial plan

cf85ab6

Copilot AI assigned Copilot and pelikhan Nov 6, 2026

Copilot started work on behalf of pelikhan November 6, 2026 08:54

View session

Copilot AI linked an issue Nov 6, 2026 that may be closed by this pull request

API proxy token usage reports 0 input and cache tokens for an Anthropic-compatible Z.AI endpoint #75271

Closed

test: preserve Z.AI Anthropic-compatible token usage totals

2d2968b

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Fix token usage reporting for Z.AI Anthropic endpoint~~ Add regression coverage for Z.AI input and cache token totals Nov 6, 2026

Copilot finished work on behalf of pelikhan November 6, 2026 09:14

Copilot AI requested a review from pelikhan November 6, 2026 09:14

github-actions Bot mentioned this pull request Nov 6, 2026

[PR Triage Report] PR triage report - 2026-11-06 #75416

Closed

pelikhan marked this pull request as ready for review November 6, 2026 13:55

Copilot AI auto review requested due to automatic review settings November 6, 2026 13:55

Copilot started reviewing on behalf of pelikhan November 6, 2026 14:00

View session

github-actions Bot reviewed Nov 6, 2026

View reviewed changes

Copilot AI reviewed Nov 6, 2026

View reviewed changes

github-actions Bot reviewed Nov 6, 2026

View reviewed changes

github-actions Bot mentioned this pull request Nov 6, 2026

[pr-sous-chef] PR Sous Chef: Processing complete #75304

Open

Copilot started work on behalf of gh-aw-bot November 6, 2026 14:27

View session

pelikhan merged commit 565bf51 into main Nov 6, 2026
173 of 174 checks passed

pelikhan deleted the copilot/fix-api-proxy-token-usage-reports branch November 6, 2026 14:48

Copilot AI requested a review from gh-aw-bot November 6, 2026 14:50

Copilot stopped work on behalf of gh-aw-bot due to an error November 6, 2026 14:50
The session was cancelled by the user.

Conversation

Copilot AI commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Design Decision Gate 🏗️

Uh oh!

github-actions Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Specification Reviewer

Uh oh!

github-actions Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Quality Sentinel

Uh oh!

github-actions Bot commented Nov 6, 2026

Comment Memory

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Nice to have

Uh oh!

github-actions Bot Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Skills-Based Review 🧠

Key Themes

Positive Highlights

Verification gap

Uh oh!

github-actions Bot Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

gh-aw-bot commented Nov 6, 2026

Uh oh!

Copilot AI commented Nov 6, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Copilot AI commented Nov 6, 2026 •

edited

Loading

github-actions Bot commented Nov 6, 2026 •

edited

Loading

github-actions Bot commented Nov 6, 2026 •

edited

Loading

github-actions Bot commented Nov 6, 2026 •

edited

Loading

github-actions Bot commented Nov 6, 2026 •

edited

Loading

github-actions Bot commented Nov 6, 2026 •

edited

Loading

github-actions Bot commented Nov 6, 2026 •

edited

Loading