Skip to content

Add auto-detected Tools and MCP Services views to the dashboard (into #354) - #355

Merged
TonsOfFun merged 5 commits into
claude/dashboard-traces-interactions-expand-6eitzffrom
claude/dashboard-tools-mcp-views-pdieob
Aug 13, 2026
Merged

Add auto-detected Tools and MCP Services views to the dashboard (into #354)#355
TonsOfFun merged 5 commits into
claude/dashboard-traces-interactions-expand-6eitzffrom
claude/dashboard-tools-mcp-views-pdieob

Conversation

@TonsOfFun

@TonsOfFun TonsOfFun commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Why

The dashboard had no answer to "what can my agents actually do?" Tools appeared only as chips inside a single trace, and MCP servers appeared nowhere at all — despite agents already calling them.

Anything built on a hand-maintained registry would have been wrong the first time someone shipped a tool without updating it. So nothing here is registered: the inventory is derived from records the engine already stores.

Third in the stack: #353#354 → this. It targets #354's branch, so this diff is only the two new views; retarget as the ones below it merge.

Where the inventory comes from

ToolDiscovery merges four sources, because each sees a different slice of the same traffic:

Source What only it can see
The offered tool roster — prompt.input.tools on traces, provenance["tools"] on solid_agent generations Tools the model was given and never called; descriptions and parameter names
Telemetry tool spans Timing, status, error rates — and agents running in a host app reporting over the wire
agent_generations.tool_calls Calls the model requested, including ones whose run died before a span existed
agent_messages with role: "tool" The arguments a tool was really invoked with, which telemetry truncates

A dashboard-executed run writes to several at once, so counting per-source would multiply-count it. Each feeds a distinct counter (calls / requested / results), telemetry stays authoritative, and installs with telemetry off fall back to the strongest signal present rather than reporting a table of zeros.

Agents' configured tools and mcp_servers are unioned in last, so "configured, never called" is visible — the difference between a tool that is broken and one that was never wired up.

MCP attribution

The mcp__server__tool convention, tagged onto spans at instrumentation time by the framework's ToolOrigin (first commit here) and re-derived on read for older traces. McpCatalog hints cover servers whose tools are exposed unnamespaced, and a dashboard builtin of the same name wins that tie, since the engine is the one running it.

Detection never depends on the catalog. A server nobody documented shows up the moment its traffic lands, flagged undocumented — those are precisely the ones worth surfacing.

The catalog, and starting a server

McpCatalog gives the MCP Services view a default set (Playwright, Filesystem, Fetch, Git, Memory, Sequential Thinking, GitHub, Slack, Postgres, and the dashboard's own /mcp), so an install sees what it could connect and not only what it already uses.

The six that need no host credentials can be started in a sandbox from the view; the rest are listed with what they would need, because there is nowhere safe to source those secrets from. SandboxSession records which servers it was launched with, so a running server reads as running instead of offering to start a second copy.

Portability and seams

  • Scopes are passed into ToolDiscovery rather than derived, so the ownership seam decides visibility — the controllers pass owned_traces / owner_agents.
  • Spans are read in Ruby, not traversed in SQL: jsonb traversal would be PostgreSQL-only. The provenance lookup takes the jsonb_exists fast path only when the adapter has it. The suite runs on SQLite.
  • builtin_tools resolves lazily; referencing an autoloaded constant during class definition would bind whatever loaded first.

Two ownership bugs this turned up

Both were found driving the engine mounted in a real multi-tenant host, and neither is reachable from the single-user dummy app:

  • SandboxSession declares owned_by :user, :account, so its owner association is :user — but current_owner is the account in multi-tenant mode. Assigning through the seam put an Account in the user association and raised AssociationTypeMismatch on every launch. Now assigned the way SandboxesController already does it: a sandbox belongs to whoever opened it.
  • respond_to?(:account=) is true whenever an account model is configured, since the association is declared from configuration — but a host app's pre-existing table may not have the column, which raised MissingAttributeError. Assignment now checks the attribute exists.

Testing

Review notes

@TonsOfFun
TonsOfFun force-pushed the claude/dashboard-tools-mcp-views-pdieob branch from c9a6ff3 to f224c15 Compare August 13, 2026 18:15
@TonsOfFun TonsOfFun changed the title Add auto-detected Tools and MCP Services views to the dashboard Add auto-detected Tools and MCP Services views to the dashboard (into #354) Aug 13, 2026
@TonsOfFun
TonsOfFun changed the base branch from claude/activeagents-local-engine-ky54n7 to claude/dashboard-traces-interactions-expand-6eitzf August 13, 2026 18:16
@TonsOfFun
TonsOfFun marked this pull request as ready for review August 13, 2026 18:47
@TonsOfFun
TonsOfFun force-pushed the claude/dashboard-tools-mcp-views-pdieob branch from f224c15 to 0774935 Compare August 13, 2026 22:36
claude added 5 commits August 13, 2026 22:38
Tool spans carried only the name a provider used to invoke the tool, so
anything reading traces had to guess where that tool came from — and every
consumer guessed separately. The dashboard's tool roster inferred an origin
one way, activeagents.ai another, and an OTLP exporter not at all.

Classify at instrumentation time instead, in one place. ToolOrigin reads the
`mcp__<server>__<tool>` convention MCP clients emit and writes the result
onto the span (tool.origin, tool.mcp_server, tool.base_name), so a trace
stays self-describing and every consumer reads the same fields.

TelemetryTrace gains the matching readers:

  - tool_usage normalizes tool spans, preferring the recorded attribution
    and falling back to name classification for traces ingested before
    tagging shipped.
  - declared_tools reads the roster the generation request offered
    (prompt.input.tools) — the only source that sees a tool the model was
    given and never called, and the only one carrying descriptions and
    parameter names.
  - mcp_servers unions the servers offered with the servers called.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BoEMwqrTweQW7rSuiJLqSG
Neither view existed, and there was no inventory to build one from: tools
showed up only as chips inside a trace, and MCP servers appeared nowhere at
all despite agents already calling them.

ToolDiscovery assembles the inventory from records the engine already
stores, so nothing needs registering by hand:

  - the tool roster in each generation request body (prompt.input.tools on
    traces, provenance["tools"] on solid_agent generations) — descriptions,
    parameters, and the tools a model was offered but never called
  - telemetry tool spans — timing, status, and error rates
  - solid_agent generation tool_calls and role:"tool" messages — the calls
    requested and the results returned
  - each agent's configured tools and mcp_servers

A dashboard-executed run writes to several of those at once, so each source
feeds a distinct counter and telemetry stays authoritative for the call
count; installs running with telemetry off fall back to the strongest
signal present instead of reporting zeros.

MCP attribution comes from the `mcp__server__tool` convention (tagged onto
spans by the framework's ToolOrigin), with McpCatalog hints covering servers
whose tools are exposed unnamespaced. Detection never depends on the
catalog — an unlisted server still appears the moment its traffic lands,
flagged undocumented.

McpCatalog also gives the MCP Services view its default set, so an install
sees what it could connect and not just what it already uses. Entries that
run without host credentials can be started in a sandbox from the view; the
rest are listed with what they'd need. SandboxSession records which servers
it was launched with, so a running server reads as running rather than
offering to start a second copy.

Scopes are passed into ToolDiscovery rather than derived, so the ownership
seam decides what is visible, and the provenance lookup takes the jsonb
fast path only on PostgreSQL — the suite runs on SQLite.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BoEMwqrTweQW7rSuiJLqSG
SandboxSession declares `owned_by :user, :account`, so its owner
association is :user wherever a host app configures a user model. But
`current_owner` is the *account* in a multi-tenant install, so assigning
through the ownership seam put an Account in the user association and
raised AssociationTypeMismatch on every launch.

Assign user and account directly, the way SandboxesController already
does — a sandbox belongs to whoever opened it. Both are skipped in a
single-user install, which declares neither association.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BoEMwqrTweQW7rSuiJLqSG
respond_to?(:account=) is true whenever the host app configures an account
model, because the association is declared from configuration. The column
behind it is a separate question: a host app mounting the engine over its
own pre-existing tables may not have one, and assigning then raises
MissingAttributeError.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BoEMwqrTweQW7rSuiJLqSG
Three defects in the two new views, found reviewing this stack.

`where.not(column: [])` does not mean "this JSON array has entries". Active
Record reads the empty array as an empty IN list and compiles the whole
condition to `1=1`, so both filters matched every row. The generations sweep
therefore scanned every generation in the window rather than the ones with
tool calls, and the running-sandbox panel listed every active sandbox —
pushing the genuinely MCP-backed ones out of its own 20-row limit, so the
view offered to start a second copy of a server that was already running.
There is no portable Active Record spelling for this, so it joins the other
per-adapter forms in AdapterAware.

Launching an MCP server provisions a sandbox and runs something in it, which
is execution: it now answers to the read-only kill switch and the quota seam,
and records usage, like every other execution path.

The inventory is derived by reading records, and the window alone does not
bound how many there are — 90 days of a busy workspace is millions of rows.
The provenance scan also ran twice per request, once for the inventory and
again for a boolean the response only uses to explain an empty table. It is
memoized, the boolean is a cheap probe on PostgreSQL, and the two large
sweeps stop at a fixed ceiling and log when they do. A precomputed summary
is the real fix; this keeps the page answering until there is one.
@TonsOfFun
TonsOfFun force-pushed the claude/dashboard-tools-mcp-views-pdieob branch from 0774935 to a59df70 Compare August 13, 2026 22:39
TonsOfFun pushed a commit that referenced this pull request Aug 13, 2026
…views-pdieob

Add auto-detected Tools and MCP Services views to the dashboard
@TonsOfFun
TonsOfFun merged commit a59df70 into main Aug 13, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants