Skip to content

fix(dashboard): collect Codex token usage - #423

Merged
jeff-r2026 merged 2 commits into
Tencent:mainfrom
linjinglan:codex/fix-codex-token-usage
Sep 8, 2026
Merged

fix(dashboard): collect Codex token usage#423
jeff-r2026 merged 2 commits into
Tencent:mainfrom
linjinglan:codex/fix-codex-token-usage

Conversation

@linjinglan

Copy link
Copy Markdown

Summary

  • Add native Codex token usage collection for Desktop token_usage_record and CLI event_msg / token_count transcripts.
  • Treat Codex totals as cumulative snapshots, wait for the final post-Stop flush, and split inclusive input/cache buckets without double counting.
  • Preserve existing prompts, token totals, and intervention stats when skill usage is merged.
  • Update the usage guide for Claude Code, CodeBuddy, and Codex token sources.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature causing existing behavior to change)
  • Documentation only
  • Refactor / internal cleanup

Test Plan

  • npm run typecheck
  • npm run build
  • npx vitest run src/__tests__/dashboard-collector.test.ts src/__tests__/conversation-token-metrics.test.ts (99 passed)
  • npx vitest run --config vitest.e2e.config.ts src/__tests__/conversation-token-e2e.test.ts (2 passed)
  • npx vitest run src/__tests__/usage-tracking.test.ts -t mergeStats (6 passed)

Full npm test on Windows completed with 2,552 passing and 91 pre-existing platform/environment failures (POSIX path, permission-mode, and bash assumptions); all tests covering the changed behavior pass.

Notes for Reviewers

Codex reports inclusive input_tokens and cumulative thread totals. The collector uses the latest cumulative snapshot and maps cached/cache-write subsets into TeamAI's disjoint buckets so the digest total is not inflated. The Stop collector runs in the existing detached background handler, so the bounded flush wait does not delay the agent UI.

@jeff-r2026
jeff-r2026 self-requested a review September 7, 2026 02:58
@jeff-r2026

Copy link
Copy Markdown
Collaborator

Blocking: the legacy Codex cumulative counter is scoped to a rollout file, not reliably to the TeamAI session.

I can reproduce this with two real Codex Desktop rollout files created when the same session was reopened/materialized again. Both files have exactly the same session_meta.payload.id, source: "vscode", originator: "Codex Desktop", and cwd:

session id: 01a05cfb-f5a6-7bb3-862a-17868df6bcb6

rollout A
  started:     2026-09-01T12:40:11.785Z
  completed:   2026-09-01T13:26:15.211Z
  first total: 21,906
  final total: 8,650,589

rollout B
  started:     2026-09-01T13:27:33.513Z
  first total: 143,674
  final total: 2,523,132

So the second file starts only 78 seconds after the first completes, retains the same session id, but its legacy event_msg/token_count/info.total_token_usage counter starts over.

Running this PR's scanTranscriptStop() against the two files produces:

{
  "first": {
    "input": 283144,
    "output": 27733,
    "cacheRead": 8339712,
    "cacheCreation": 0
  },
  "resumed": {
    "input": 29828,
    "output": 2936,
    "cacheRead": 2490368,
    "cacheCreation": 0
  }
}

Feeding those two Stop snapshots, in timestamp order, through aggregateSessionMetrics() leaves only the second snapshot because the code assumes "latest wins":

{
  "latestWins": {
    "input": 29828,
    "output": 2936,
    "cacheRead": 2490368,
    "cacheCreation": 0
  }
}

More importantly, if rollout A has already been reported, computePromptTokenDelta() reports no tokens at all for rollout B:

{
  "deltaAfterResume": {
    "input": 0,
    "output": 0,
    "cacheRead": 0,
    "cacheCreation": 0
  }
}

This silently drops the first 2.5M tokens from the reopened segment (and temporarily makes the dashboard regress from 8.65M to 2.52M). A new prompt can then advance the persisted baseline to the smaller snapshot, making that omitted segment permanent.

This grouping is consistent with Codex itself: the Stop hook receives session_id from sess.session_id(), while transcript_path is resolved from the current rollout path: https://github.com/openai/codex/blob/1fb5158b3496a05abb89fb992d45737a02511d47/codex-rs/core/src/hook_runtime.rs#L377-L444

I think the legacy event_msg.total_token_usage snapshot needs a rollout/segment scope (the existing transcriptPath can identify it), with the latest snapshot retained per segment and the segment totals summed. This should be kept distinct from the newer token_usage_record.thread_token_usage semantics so thread-level records are not double-counted.

A regression test should use one sessionId with two different transcript paths where the second cumulative counter restarts below the first; the expected session/report total must include both segments and remain idempotent when either Stop is scanned again.

林景蓝 added 2 commits September 8, 2026 10:56
- parse Codex Desktop and CLI cumulative token records
- wait for post-Stop usage flush without double counting cache tokens
- preserve existing session metrics when merging usage stats
- distinguish thread-level token_usage_record from rollout-scoped token_count totals
- sum the latest legacy snapshot per transcript and order background Stops by event time
- add resume, mixed-format, delta, and idempotency regression coverage
- document Codex snapshot semantics
@linjinglan
linjinglan force-pushed the codex/fix-codex-token-usage branch from 0fa92be to 8c3a0d1 Compare September 8, 2026 03:04
@jeff-r2026
jeff-r2026 merged commit 49ffdf9 into Tencent:main Sep 8, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants