feat: render inputs, latency, and token usage in agent test results @W-23524159@ - #481
Open
niharpatel-salesforce wants to merge 10 commits into
Conversation
Scopes the CLI-only rendering of PROMPT test-case inputs, latency, and token usage in `sf agent test results` human format.
Two-task plan: inputs line, then latency/tokens line, both in humanFormatAgentforceStudio human-readable output only.
Bring PR salesforcecli#481 up to date with upstream main (agents 2.0.6 bump, AFS nut test hardening, plugin-agent 2.0.5 release).
These design/plan markdown files were an artifact of my own tooling defaults, not a repo convention (no prior PR has anything like them) - removing them so the PR only carries the actual code change. The design/plan content lives in the PR description instead.
inputs[].name is an API field name, not a display label - show it verbatim instead of title-casing it.
| label: 'TAP', | ||
| get: () => convertAgentforceStudioTestResultsToFormat(results, 'tap'), | ||
| strip: false, | ||
| }, |
There was a problem hiding this comment.
This hunk is a pure Prettier auto-format fix - no logic change.
This repo's config sets printWidth: 120;
The original single-line json/junit/tap entries were 125–131 chars each (over the limit), while human was 115 chars (under it), which is why only those three got wrapped and human was left alone.
Confirmed the original file already failed prettier --check at this exact spot before this PR.
niharpatel-salesforce
marked this pull request as ready for review
August 27, 2026 12:27
…zation getTestCaseInputs filtered test-case inputs to string-only values, silently dropping any number/boolean value and, when every value was non-string, wrongly falling back to the legacy User Input line. Also replaces the anonymous `as unknown as` cast on the SDK-undeclared `inputs` field with a named, documented type alias. Remote API strings (inputs[].name/value, subjectResponse.userInput) were interpolated into the table title unsanitized on the default interactive output path, letting a malicious payload inject terminal escape sequences - stripVTControlCharacters was only ever applied on the --output-dir file-write path. Sanitize the raw values before rendering, preserving the code's own ansis coloring, and collapse embedded newlines since stripVTControlCharacters doesn't touch plain newlines. Also parses subjectResponse once per test case instead of twice.
PROMPT-subject scorers often only populate reasoning, leaving Expected and Actual permanently blank in the scorer table. Show those two columns only when at least one scorer row in that test case has a value for either - they're a paired unit, so both show or neither does, and the decision is made independently per test case rather than once for the whole result set.
stripVTControlCharacters only removes ANSI/VT escape sequences, and the existing newline-collapse regex only fired when a `\n` was present. A value containing a bare `\r` with no trailing `\n` (or a stray `\v`, `\f`, BEL, backspace) passed through untouched. A lone `\r` returns the cursor to column 0 and overwrites the already-rendered line, so an input value like `safe\rInputs: EVIL` could rewrite what the operator reads on the terminal. Collapse the full whitespace-control set alongside newlines, then strip any remaining C0/DEL bytes outright.
nico-pappagianis
approved these changes
Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Renders per-test-case Inputs and Latency/Tokens in the human-readable output of
sf agent test results, for AgentforceStudio (NGT) results — the format currently shared by bothAGENTandPROMPTsubject types.getTestCaseInputs,formatInputsLineinsrc/handleTestResults.ts): reads the per-test-caseinputs: [{name, value}]array, which exists on the wire (confirmed by tracing the SDK'snormalizeAgentforceStudioResults, which preserves it via object spread) but isn't declared on theAgentforceStudioTestCaseResulttype in@salesforce/agents— read via a locally type-guarded cast, no upstream type change. Falls back to the existingsubjectResponse.userInputparsing wheninputsis absent, so anything relying on that today is unaffected. Field names are shown exactly as returned by the API (no re-labeling/title-casing —inputs[].nameis an API field name, not a display label). Shows the first 3 inputs,(+N more)beyond that.parseSubjectResponseMetrics,formatMetricsLine): parsesperformance.latency.durationandtokenUsage.{completion, prompt.total, total}out of thesubjectResponseJSON string. Renders only the parts that exist; omits the line entirely when neither is present.subjectTypefield, so rendering triggers purely on whether the underlying data exists, for either subject type.@salesforce/agents, no Connect API / results-contract changes. Scoped tosrc/handleTestResults.tsplus tests and fixtures.AGENT(result + failure reasoning), with no inputs/latency/tokens; this PR only touches the human-readable format.humanFormatAgentforceStudio, which had none before this change: 8 new tests covering inputs-present, legacy-userInput-fallback, graceful omission when no data is present, 4+ inputs truncation, latency-only, tokens-only, and the combined case. Full suite: 455/455 passing, no regressions.sf agent test results --test-runner agentforce-studio) —User Inputrenders unchanged, no spuriousInputs/Latency/Tokenslines appear since AGENT'ssubjectResponsedoesn't carry those fields today.Screenshot of the rendered output

What issues does this PR fix or reference?
@W-23524159@