Add auto-detected Tools and MCP Services views to the dashboard (into #354) - #355
Merged
TonsOfFun merged 5 commits intoAug 13, 2026
Conversation
TonsOfFun
force-pushed
the
claude/dashboard-tools-mcp-views-pdieob
branch
from
August 13, 2026 18:15
c9a6ff3 to
f224c15
Compare
TonsOfFun
changed the base branch from
claude/activeagents-local-engine-ky54n7
to
claude/dashboard-traces-interactions-expand-6eitzf
August 13, 2026 18:16
TonsOfFun
marked this pull request as ready for review
August 13, 2026 18:47
TonsOfFun
force-pushed
the
claude/dashboard-tools-mcp-views-pdieob
branch
from
August 13, 2026 22:36
f224c15 to
0774935
Compare
Tool spans carried only the name a provider used to invoke the tool, so
anything reading traces had to guess where that tool came from — and every
consumer guessed separately. The dashboard's tool roster inferred an origin
one way, activeagents.ai another, and an OTLP exporter not at all.
Classify at instrumentation time instead, in one place. ToolOrigin reads the
`mcp__<server>__<tool>` convention MCP clients emit and writes the result
onto the span (tool.origin, tool.mcp_server, tool.base_name), so a trace
stays self-describing and every consumer reads the same fields.
TelemetryTrace gains the matching readers:
- tool_usage normalizes tool spans, preferring the recorded attribution
and falling back to name classification for traces ingested before
tagging shipped.
- declared_tools reads the roster the generation request offered
(prompt.input.tools) — the only source that sees a tool the model was
given and never called, and the only one carrying descriptions and
parameter names.
- mcp_servers unions the servers offered with the servers called.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BoEMwqrTweQW7rSuiJLqSG
Neither view existed, and there was no inventory to build one from: tools
showed up only as chips inside a trace, and MCP servers appeared nowhere at
all despite agents already calling them.
ToolDiscovery assembles the inventory from records the engine already
stores, so nothing needs registering by hand:
- the tool roster in each generation request body (prompt.input.tools on
traces, provenance["tools"] on solid_agent generations) — descriptions,
parameters, and the tools a model was offered but never called
- telemetry tool spans — timing, status, and error rates
- solid_agent generation tool_calls and role:"tool" messages — the calls
requested and the results returned
- each agent's configured tools and mcp_servers
A dashboard-executed run writes to several of those at once, so each source
feeds a distinct counter and telemetry stays authoritative for the call
count; installs running with telemetry off fall back to the strongest
signal present instead of reporting zeros.
MCP attribution comes from the `mcp__server__tool` convention (tagged onto
spans by the framework's ToolOrigin), with McpCatalog hints covering servers
whose tools are exposed unnamespaced. Detection never depends on the
catalog — an unlisted server still appears the moment its traffic lands,
flagged undocumented.
McpCatalog also gives the MCP Services view its default set, so an install
sees what it could connect and not just what it already uses. Entries that
run without host credentials can be started in a sandbox from the view; the
rest are listed with what they'd need. SandboxSession records which servers
it was launched with, so a running server reads as running rather than
offering to start a second copy.
Scopes are passed into ToolDiscovery rather than derived, so the ownership
seam decides what is visible, and the provenance lookup takes the jsonb
fast path only on PostgreSQL — the suite runs on SQLite.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BoEMwqrTweQW7rSuiJLqSG
SandboxSession declares `owned_by :user, :account`, so its owner association is :user wherever a host app configures a user model. But `current_owner` is the *account* in a multi-tenant install, so assigning through the ownership seam put an Account in the user association and raised AssociationTypeMismatch on every launch. Assign user and account directly, the way SandboxesController already does — a sandbox belongs to whoever opened it. Both are skipped in a single-user install, which declares neither association. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BoEMwqrTweQW7rSuiJLqSG
respond_to?(:account=) is true whenever the host app configures an account model, because the association is declared from configuration. The column behind it is a separate question: a host app mounting the engine over its own pre-existing tables may not have one, and assigning then raises MissingAttributeError. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BoEMwqrTweQW7rSuiJLqSG
Three defects in the two new views, found reviewing this stack. `where.not(column: [])` does not mean "this JSON array has entries". Active Record reads the empty array as an empty IN list and compiles the whole condition to `1=1`, so both filters matched every row. The generations sweep therefore scanned every generation in the window rather than the ones with tool calls, and the running-sandbox panel listed every active sandbox — pushing the genuinely MCP-backed ones out of its own 20-row limit, so the view offered to start a second copy of a server that was already running. There is no portable Active Record spelling for this, so it joins the other per-adapter forms in AdapterAware. Launching an MCP server provisions a sandbox and runs something in it, which is execution: it now answers to the read-only kill switch and the quota seam, and records usage, like every other execution path. The inventory is derived by reading records, and the window alone does not bound how many there are — 90 days of a busy workspace is millions of rows. The provenance scan also ran twice per request, once for the inventory and again for a boolean the response only uses to explain an empty table. It is memoized, the boolean is a cheap probe on PostgreSQL, and the two large sweeps stop at a fixed ceiling and log when they do. A precomputed summary is the real fix; this keeps the page answering until there is one.
TonsOfFun
force-pushed
the
claude/dashboard-tools-mcp-views-pdieob
branch
from
August 13, 2026 22:39
0774935 to
a59df70
Compare
TonsOfFun
pushed a commit
that referenced
this pull request
Aug 13, 2026
…views-pdieob Add auto-detected Tools and MCP Services views to the dashboard
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The dashboard had no answer to "what can my agents actually do?" Tools appeared only as chips inside a single trace, and MCP servers appeared nowhere at all — despite agents already calling them.
Anything built on a hand-maintained registry would have been wrong the first time someone shipped a tool without updating it. So nothing here is registered: the inventory is derived from records the engine already stores.
Third in the stack: #353 → #354 → this. It targets #354's branch, so this diff is only the two new views; retarget as the ones below it merge.
Where the inventory comes from
ToolDiscoverymerges four sources, because each sees a different slice of the same traffic:prompt.input.toolson traces,provenance["tools"]on solid_agent generationsagent_generations.tool_callsagent_messageswithrole: "tool"A dashboard-executed run writes to several at once, so counting per-source would multiply-count it. Each feeds a distinct counter (
calls/requested/results), telemetry stays authoritative, and installs with telemetry off fall back to the strongest signal present rather than reporting a table of zeros.Agents' configured
toolsandmcp_serversare unioned in last, so "configured, never called" is visible — the difference between a tool that is broken and one that was never wired up.MCP attribution
The
mcp__server__toolconvention, tagged onto spans at instrumentation time by the framework'sToolOrigin(first commit here) and re-derived on read for older traces.McpCataloghints cover servers whose tools are exposed unnamespaced, and a dashboard builtin of the same name wins that tie, since the engine is the one running it.Detection never depends on the catalog. A server nobody documented shows up the moment its traffic lands, flagged
undocumented— those are precisely the ones worth surfacing.The catalog, and starting a server
McpCataloggives the MCP Services view a default set (Playwright, Filesystem, Fetch, Git, Memory, Sequential Thinking, GitHub, Slack, Postgres, and the dashboard's own/mcp), so an install sees what it could connect and not only what it already uses.The six that need no host credentials can be started in a sandbox from the view; the rest are listed with what they would need, because there is nowhere safe to source those secrets from.
SandboxSessionrecords which servers it was launched with, so a running server reads as running instead of offering to start a second copy.Portability and seams
ToolDiscoveryrather than derived, so the ownership seam decides visibility — the controllers passowned_traces/owner_agents.jsonb_existsfast path only when the adapter has it. The suite runs on SQLite.builtin_toolsresolves lazily; referencing an autoloaded constant during class definition would bind whatever loaded first.Two ownership bugs this turned up
Both were found driving the engine mounted in a real multi-tenant host, and neither is reachable from the single-user dummy app:
SandboxSessiondeclaresowned_by :user, :account, so its owner association is:user— butcurrent_owneris the account in multi-tenant mode. Assigning through the seam put anAccountin the user association and raisedAssociationTypeMismatchon every launch. Now assigned the waySandboxesControlleralready does it: a sandbox belongs to whoever opened it.respond_to?(:account=)is true whenever an account model is configured, since the association is declared from configuration — but a host app's pre-existing table may not have the column, which raisedMissingAttributeError. Assignment now checks the attribute exists.Testing
OPENAI_API_KEYones, the same count Extract the dashboard into the actionagent gem, with feature parity #353 reportsReview notes
actionagent/app/assets/builds/action_agent.jsis rebuilt — committed on purpose so host apps skip the JS build. It was the only rebase conflict against One expandable-object design for traces, spans and interactions (into #353) #354, and is regenerated rather than merged.mcp_serverscolumn is added to the install generator's table template and to the dummy app's schema. Hosts mounting over pre-existing tables need their own migration (the platform's is Rate limiting options per agent or per action #104).provenance["tools"]source stays inert until solid_agent ships Persist the offered tool roster on each generation's provenance solid_agent#5; the other three cover the feature meanwhile.TelemetryTracereaders gained the JSON-Schema parameter flattening andllm.toolsalias that had been living in a platform-side concern — there is one implementation now rather than two.