Skip to content

feat(cli): filter usage by billing route and mode - #1486

Open
ulises-c wants to merge 5 commits into
getagentseal:mainfrom
ulises-c:feat/1451-billing-route-filters
Open

ulises-c wants to merge 5 commits into
getagentseal:mainfrom
ulises-c:feat/1451-billing-route-filters

Conversation

@ulises-c

@ulises-c ulises-c commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Summary

Part of #1451. Builds on #1448, which resolved #1450. This PR is the first call-level CLI slice; it does not close #1451.

  • Restore optional billing: 'metered' | 'subscription' provenance on provider, parsed, and cached calls. Hermes included is subscription-covered, actual (including explicit $0) is metered, registered routes supply a default only when the provider recorded no billing fact, and everything else stays unknown.
  • Register the exact usage-bearing provider fields openrouter as the OpenRouter metered route and OpenCode's amazon-bedrock as the Bedrock metered route. Preserve them from Hermes billing_provider and OpenCode providerID across legacy SQLite, v2 session_message, file storage, and session-rollup fallbacks, with no speculative aliases. Bedrock remains the only id-shape route.
  • Add call-level --route direct|bedrock|openrouter and --billing metered|subscription filters to models, sessions, export, and audit. Mixed sessions contribute only matching calls; rebuilt totals retain session linkage plus existing agentName / agentStartedAt metadata.
  • Preserve raw-model pricing and Bedrock profile SKUs. Filtered $0 route/subscription rows remain visible by default; an explicit --min-cost still wins. direct is documented as direct-or-unknown, and unknown billing matches neither billing filter.
  • Repair the narrow pre-upgrade Hermes v3 ledger shape that could have stored an included zero-cost baseline as actual, without relabeling genuine multi-observation histories.

Still open in #1451: today/report/status, compare, menubar payload fields, daily-cache v34, and native/web UI. Those need durable structured route/billing aggregates. sessions --by-work-unit with either new filter is rejected before parsing rather than inferring filtered-out roots.

Testing / Verification

  • I have tested this locally against real data (not just unit tests)
  • npm test passes
  • npm run build succeeds
  • Not a UI change
  • UI change: before and after screenshots attached below

Exact base/branch comparison on Node 22:

  • Base 0f5a3345: 4,278 passed / 5 skipped / 0 failed.
  • Branch 7d6f573e: 4,324 passed / 5 skipped / 0 failed — 46 added tests, zero new failures.
  • npm run test:locks: 36 passed on base and branch.
  • npx tsc --noEmit: clean.
  • npm run build: CLI and dashboard built.
  • npm run verify:upgrade: passed (1 warm-run observation skipped by the harness).
  • Semgrep hot-path rule over src/providers/ and src/parser.ts: 0 findings / 0 errors.
  • Independent Codex review: the original implementation passed four adversarial reviews; the OpenCode extension's first review found two P2 cache/normalization issues, both were fixed test-first, and final OpenRouter and Amazon Bedrock reviews reported no regressions or blocking issues.

Real data, this machine:

main:
cohere/north-mini-code:free  route=null        cost=$0
openrouter/free             route=null        cost=$0

branch, --route bedrock (today):
GPT-5.6 Sol (Bedrock)        route=bedrock     calls=22  cost=$187.778681

branch, --route openrouter (today):
north-mini-code:free (OpenRouter)  route=openrouter  calls=1  cost=$0
free (OpenRouter)                  route=openrouter  calls=1  cost=$0

OpenCode 1.18.31, isolated cold and warm caches:
cohere/north-mini-code:free  route=openrouter  calls=3  cost=$0
openai.gpt-5.6-sol          route=bedrock     calls=1  cost=$0.1261678
amazon.nova-2-lite-v1:0     route=bedrock     calls=1  cost=$0.0024605
--route direct excludes all three routed rows
--billing metered includes all three routed rows

branch, --billing subscription (30 days):
61 provider-recorded included calls across 8 rows, recorded cost=$0

The first two OpenRouter rows come from two fresh Hermes sessions against https://openrouter.ai/api/v1: one openrouter/free coding task that read src/models.ts (4 upstream calls, 107,456 input / 668 output tokens) and one explicit cohere/north-mini-code:free request (15,001 input / 69 output tokens). Both were zero-cost.

OpenCode 1.18.31 was then run independently with openrouter/cohere/north-mini-code:free against this PR's src/models.ts. Its on-disk messages recorded providerID=openrouter, modelID=cohere/north-mini-code:free, three assistant calls, and zero reported cost. Before the extension CodeBurn parsed those calls as route=null; after it, fresh isolated cold and warm caches both return route=openrouter, --route direct excludes them, and --billing metered includes them.

Two OpenCode Amazon Bedrock sessions were also run through the AWS SDK credential chain in us-east-1. amazon-bedrock/openai.gpt-5.6-sol recorded providerID=amazon-bedrock, modelID=openai.gpt-5.6-sol, one call, and $0.1261678; CodeBurn classified it as Bedrock by both provider evidence and the existing model shape. amazon-bedrock/amazon.nova-2-lite-v1:0 recorded the same provider field with a Nova id that the model-shape detector deliberately does not recognise; this proves the new provider-field path itself. Cold and warm caches classify both as route=bedrock / billing=metered, and --route direct excludes both.

The regression suite covers exact openrouter and amazon-bedrock route detection with negative aliases, OpenCode legacy SQLite/v2/file/session-rollup storage, OpenCode and KiloCode cache invalidation across both route additions, fallback model normalization, billing precedence, Hermes cost-basis transitions and vanished sources, session-cache validation/round trips, mixed-call totals and metadata, midnight re-anchoring, all four CLI surfaces, validation-before-export, resident serve allowlists, and the legacy v3 included-baseline repair.

For new providers only:

Not applicable: this adds billing-route provenance to existing Hermes and OpenCode providers; it does not register a new coding-harness provider.

Affiliation: none. I use Bedrock and OpenRouter alongside fixed-fee coding subscriptions and need their billing paths to remain distinct.

Carry provider-observed billing provenance through Hermes and the session cache, register the real OpenRouter route, and add call-level route/billing filters to models, sessions, export, and audit.

Keep unknown billing unclassified, preserve existing pricing, and repair legacy included Hermes ledger baselines without rewriting genuine multi-observation history.
Carry OpenCode providerID through legacy SQLite, v2, file, and session-rollup paths so OpenRouter calls remain distinct from direct traffic. Invalidate OpenCode and KiloCode caches and cover strict route matching.
ozymandiashh
ozymandiashh previously approved these changes Sep 18, 2026

@ozymandiashh ozymandiashh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the full diff plus the follow-up OpenCode provenance commit. Approving from my side — the PR is merge-ready whenever you flip it out of draft.

What I verified against the corpus conventions:

  • Every claim in the body maps to code: billing provenance plumbed ParsedProviderCall → ParsedApiCall → CachedCall → turns; --route/--billing on models/sessions/export/audit with the assertRoute/assertBilling mirroring of assertProvider; call-level slicing with rebuilt totals; the Hermes v3 ledger repair. Deferrals are stated and genuinely absent, not half-done.
  • The route registration stays exact-literal through routeFromProviderField — the negative tests ('OpenRouter', ' openrouter ', model-id prefixes) hold, and the new OpenCode providerID → route passthrough uses the same helper rather than a second mapping, so no alias-without-evidence creeps in.
  • Warm-cache invalidation is the intended mechanism (PROVIDER_PARSE_VERSIONS bump) and upgrade-path CI confirms it; the unfiltered path returns the input by identity, new payload fields are additive, and the one visible output change — a separate (OpenRouter) row for Hermes OpenRouter sessions — is the feature per #1448's row-key design.

One non-blocking note I'd keep in mind: carryLinkageFields now also carries agentName/agentStartedAt, which slightly widens that function beyond the PR's stated scope (benign — fields previously dropped are now preserved — but worth a line in the body if you get a chance).

Relative to #1451 this lands the call-level CLI slice (4 of 6 commands, openrouter registration, billing-mode provenance); the payload/tab-strip half stays open as you scoped it.

Map OpenCode's exact amazon-bedrock providerID to the existing metered Bedrock route, including model families that cannot rely on Anthropic/OpenAI ID shapes. Invalidate OpenCode and KiloCode caches across the route expansion.
@ulises-c

Copy link
Copy Markdown
Contributor Author

@ozymandiashh Thanks for the review. I addressed the non-blocking scope note by stating explicitly in the PR summary that filtered rebuilds preserve the existing agentName / agentStartedAt metadata.

Since your review at 944a6962, the current head 7d6f573e adds the exact providerID=amazon-bedrock mapping based on two real OpenCode/AWS sessions. One used openai.gpt-5.6-sol; the second used Nova specifically to prove the provider-field path without relying on the Anthropic/OpenAI model-id detector. OpenCode and KiloCode cache fingerprints now invalidate across both provider-field additions, and the OpenCode provider documentation records the contract.

Current verification: 4,324 local tests plus 36 serial lock tests passed; build and upgrade verification passed; the final GitHub run has 19 successful checks, one expected skipped release-asset check, and no failures or pending jobs. Independent reviews of the post-review OpenCode and Amazon Bedrock deltas found no remaining issues.

The PR body now says Part of #1451, Builds on #1448, which resolved #1450, and explicitly leaves the payload/UI work open. Please re-review the current head when convenient; your earlier approval was dismissed automatically by the subsequent pushes.

@ulises-c
ulises-c marked this pull request as ready for review September 18, 2026 22:44
@ulises-c
ulises-c requested a review from iamtoruk as a code owner September 18, 2026 22:44
ozymandiashh
ozymandiashh previously approved these changes Sep 18, 2026

@ozymandiashh ozymandiashh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the two commits pushed after my earlier approval (dismissed as stale): the amazon-bedrock registration follows the same exact-literal discipline as openrouter — evidence-backed spelling, no invented aliases, case/whitespace normalization kept only for Hermes' shipped bedrock — and the billing-routes-v2 parse-version bump invalidates both the shared OpenCode/Kilo cache and pre-merge builds of this PR, which is the right warm-cache treatment. Docs and the route/cache-invalidation tests cover it. Still approving; from my side this is merge-ready whenever you are.

getagentseal#1491 moved the models command's floor and --top slice from
aggregateModels to the call site (unfiltered fetch + local floor + the
unpriced disclosure line). This merge keeps that fetch and applies the
PR's own rule at the floor: --route/--billing is an explicit request
for those rows, so the noise floor lifts for them too, and the
disclosure line only prints when a floor actually dropped something.

@ozymandiashh ozymandiashh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge-conflict resolution reviewed: keeps #1491's unfiltered fetch, local floor, slice and disclosure line, and lifts the floor for --route/--billing exactly as this PR intended. 147 tests green locally across this PR's suites (billing-route e2e, opencode routes, cache invalidation) plus the #1420 suite. Approving again after the push.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants