Repository navigation
feat(agentic): realistic agentic audit demo with real tool execution - #55
Draft
SushantGautam wants to merge 40 commits into
Draft
SushantGautam wants to merge 40 commits into
SushantGautam wants to merge 40 commits into
Conversation
The embedded-mode wheel pin in chat/proxy.py still pointed at the v0.11.4.3-subpath release even though the docker image default and the subpath docs (deploy/openwebui-subpath) already track v0.12.0. Align the embedded CLI-uvicorn launch with the published v0.12.0-subpath wheel so both modes serve the same build, and update the config comment + test assertion to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
fix(chat): bump bundled Open WebUI pin to v0.12.0-subpath
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…atalog v3) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…emo run harness Run-detail page now leads with objective/status/explanation (agentic_overview), a full-width chronological trajectory timeline with expandable tool args/results and KB provenance rows (agentic_timeline), prioritized evidence-linked finding cards (agentic_findings), a semantic-judge panel with span-cited rationale (agentic_judge), and collapsible raw JSON (agentic_raw). Blocks self-hide when data is absent. Responsive at >=1280 and 390px; e2e smoke extended. New management command run_demo_audits drives live audits via HTTP against the dev stack: creates runs (max_turns/n_repetitions), waits for completion, records verdict/findings/spans to results/demo_runs/*.jsonl and a markdown scorecard in docs/agentic-demo-results.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…typing Sections (tools, retrieval, trajectory, policy, budgets, approvals, handoffs, guardrails, rerank, semantic_judge, state) are now omittable; absence means 'not checked', explicit false/0/null stay assertions. Supplied fields are strictly type-checked; catalog v3 metadata validates. Template is a minimal skeleton instead of dozens of dormant defaults. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tool set Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Presets (tool accuracy, retrieval provenance, forbidden actions, semantic grounding, recovery budget) emit only the sections relevant to the audit focus; adders layer single constraints without dormant defaults. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…hash inputs Primary pack now mirrors agentic_demo_catalog_v3.json core scenarios exactly (metadata byte-identical), documents=[] everywhere, negative controls documented, no artificial constraints. D11/D12 omitted until prerequisites exist. seed_agentic_scenarios passes severity_ceiling, documents and file_uri into scenario_revision_hash. Old A01-A08 full-field reference pack removed (superseded; frozen preloaded demo fixture keeps its historical keys). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ory adders Controls form renders each category only when active; 'Add constraint' rows enable a category on demand; cleared fields drop their assertions. Merge (server + JS preview) rebuilds owned sections from scratch so stale values don't leak, keeps checkbox sentinels as explicit false, and round-trips unknown imported fields. New fields: tools.allowed, semantic_judge.enabled/criteria. Tests cover sparse create, section clearing, and unknown-field round-trip. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ication Retrieval: drop the single-linked-KB default inference — a retrieval call is attributed to a source only when the trace carries an observable KB/file/source identifier (snapshot external_id, list_knowledge file mapping, or exact scenario-entry match). Otherwise source identity is UNKNOWN and retrieval.source_allowed stays INCONCLUSIVE. Tools: OWUI runtime builtins (search_chats, get_current_timestamp) no longer count against domain-tool expectations/allowlists in tool_selection; explicit forbidden entries still catch them via tool.forbidden. tool_permissions already excluded unmapped builtins from tool.known (unchanged). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ntic evaluation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ification
- _push_to_openwebui now returns a status dict with per-asset
{asset, external_id, verified} entries and an overall healthy flag
- KB reused by name; docs re-uploaded only when content hash prefix
does not match the already-linked file id (hash-gated upload)
- Tool reused by stable owui_tool_id
- Agent links reconciled to full expected ID set via
update_workspace_model, then verified by fetching the live model
- seed_demo_agent passes existing agent row into push so link
reconciliation runs on re-runs; status is 'degraded' (not silent
skip) when any mandatory asset is missing or unverifiable
- DEMO_AGENT docs extended to all 7 committed fixture files
- Tests: idempotent second run makes zero create/upload calls;
changed doc triggers exactly one re-upload; new asset created+linked;
verification mismatch surfaces as unhealthy/degraded status
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…es, ordering Fixes exposed by the new tests: - semantic_judge: malformed per-dimension verdicts (unrecognized status) now carry a parse-failure rationale instead of an empty one. - retrieval: direct data_source values that do not resolve through the snapshot external_id or list-knowledge file mapping are no longer attributed — a spoofable display name alone can never PASS. Unresolvable identifiers stay INCONCLUSIVE (never FAIL). test_pr2_pr8 updated to keep its FAIL case resolvable via a two-KB snapshot. Pinned-as-is: bounded evidence packet (25 steps, <1KB fields), builtin-name collision precedence in tool_selection/tool_permissions, deterministic first-occurrence tie-breaks for identical repeated steps. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Lane 4's findings cards replaced the legacy check list, which broke the verify_agentic_audit ui-evidence-linking contract (check.evidence_span_ids rendered with span jump links). Restored the raw check-details block under the prioritized cards and exposed evidence_span_ids on finding view data. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…view Lane 4's richer agentic overview card reworded the one-line rollup, breaking the preloaded-demo test contract. Restored the canonical marker alongside the new layout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…lthy OWUI rejects re-upload of identical bytes with 400 'Duplicate content detected'; the reconciler marked every unchanged doc unverified on each re-run (healthy=False). Now a linked file is verified via its stored content hash when available, and a duplicate rejection for an already-linked doc is treated as verified. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…havior
Live probing of the running OWUI (v0.12 subpath build) showed: KB detail
returns files:null, the files listing endpoint is empty for this account, and
file/add dedupes by content hash ('Duplicate content detected' 400). The
reconciler now treats that rejection as store-level content verification and
records it honestly; KB + tool + agent links verify by live read-back.
healthy=True achieved on the live stack.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…_model_id alongside judge_id Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… key absent - run the agentic judging lane when scenario metadata declares it, not only for agent-targeted runs (bare-model targets still emit tool/retrieval spans) - drop target-only 'tools' gen param from auditor/judge requests (gateways reject it with HTTP 400) - surface the last assistant turn as judgment.final_response so the evidence packet carries the response being judged - pass the scenario's declared trace.content_capture into span collection (credential default stays the floor) - omit semantic_judgement entirely when the judge is disabled so consumers can tell 'not configured' from an executed UNCERTAIN judgement - live probe script (scripts/probe_semantic_judge.py) for judge resolution - fix stale expectation in chat KB file listing test (hash field) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…io params to OpenAI extra_body (session_id/tool_ids for OWUI's saved-chat path) was dropped on the repeated-reps execution path, so agent turns never reached the server-side tool-execution chat. auditor_kwargs now extracts it from the merged target param dicts and hands it back: build_model_auditor injects it into target_params, TraceContextModelAuditor does the same for the rep runner. _streaming_call also pops extra_headers (W3C traceparent) before calling the OpenAI client — a studio-level parameter that previously leaked as an unknown request field — while extra_body is intentionally kept so any_llm forwards it to the compat endpoint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…el runs When a run has no explicit system prompt, augment_agent_generation now injects the frozen agent's system prompt. This lets a bare-model target (e.g. SimulaChat directly, without an OpenWebUI wrapper) execute as the audited agent instead of a generic assistant. An explicit run-level system prompt always wins. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…model The managed "Open WebUI Agents" RegisteredModel reference is a pointer, not a second identity. When its model_id no longer matches the agent's base model (e.g. the OWUI agent model was removed or re-pointed), _ensure_agent_ model_reference now re-points it so agent-targeted runs execute against a live endpoint instead of failing on a stale one. The lookup also finds the managed reference by connection rather than exact model_id match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Agent base-model override: execute against the agent's resolved base
model connection (SimulaChat) instead of the OWUI wrapper endpoint,
using the correct API key from that connection.
- Tool injection: register the agent's tools as OpenAI function-calling
definitions in target_params so bare-model targets can emit tool_calls.
Supports multi-function OWUI tools (list of function schemas).
- Streaming tool-call capture: _streaming_call now accumulates tool_call
deltas from OpenAI-compatible streams and serializes them into content
as [TOOL_CALL name(args)] markers for downstream detection.
- Conversation fallback trajectory: when no OTLP spans are captured
(bare-model targets), derive tool-call steps from the recorded
conversation so deterministic checks (tool.required, tool.arguments)
can still evaluate.
- Pydantic-safe field access in streaming parser (handles both dict and
object responses from any_llm/OpenAI clients).
Verified: D01 scenario now shows tool.required PASS with
acme_lookup_order({"order_id": "ACME-1001"}) detected.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…remove leaked template comments _agentic_timeline_view synthesizes minimal span dicts from conversation TOOL_CALL markers when no OTLP spans exist, so evidence links resolve to visible timeline rows. trajectory.from_conversation handles both streaming and non-stream marker formats. Remove Django template comments that were leaking into rendered HTML. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…overview The results section is directly below the agentic audit card, making the anchor link unnecessary clutter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Checks with no expected/observed values and no evidence spans render as plain rows instead of empty <details> elements with a misleading arrow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move the trace-span expand script outside the rep.trace.spans condition so it works for conversation-derived spans too. Also handle direct hash navigation (URL with #trace-span-...) not just in-page clicks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The anchor ID is on the <li> but <details> is its child, not parent. closest() walks up the tree and missed it. Now try querySelector (descendants) first, then closest (ancestors) as fallback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…e link click Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The section showed a confusing 'no spans exported' message when the target (SimulaChat) doesn't emit OTLP traces. Now it only renders when actual spans are present. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When the model returns tool_calls, Studio now executes the tools locally (deterministic in-memory functions from the demo fixture), appends the results as role=tool messages, and re-calls the model until it produces a text answer or the tool-loop budget is exhausted. This gives us real multi-step agent trajectories: user → model calls tool → tool executes → model answers with data Verified live (run #17, trace 117d6210): D01 PASSES with the model calling acme_lookup_order(ACME-1001), receiving the order data, and responding with the correct items and delivery date. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
SushantGautam
force-pushed
the
agents/agentic-audit-realistic-demo
branch
from
October 15, 2026 15:08
9d8a10f to
7e14f40
Compare
Add keyword-based policy knowledge search to the Acme Tools class. The agent can now retrieve relevant policy passages from the local fixture documents (refunds, returns, shipping, exceptions, tracking). - Add acme_search_knowledge method to order_lookup.py Tools class - Register as Studio Tool (external_id=acme_search_knowledge) - Update _build_tool_executor to recognize the new tool name - Fix missing json/inspect imports in trace_target.py - Update agent system prompt to mention KB search capability Verified: D01 PASS (run #23), D03 executes correctly (run #22) with real tool call + result + factual answer. D03's retrieval.required check fails because the tool already provides refund_window_days — a scenario calibration issue, not an execution bug. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Key fixes: - Trajectory: capture locally-executed tools from conversation markers ([TOOL_CALL]/[TOOL_RESULT]) and merge into OTLP-derived trajectory - Retrieval checks: recognize acme_search_knowledge as retrieval tool, add KB filename→name resolution fallback for source_allowed check - Agent sync: fix _ensure_agent_model_reference to use external_id (not base_model_id) so agent_target_model() can find the reference - Tool redesign: replace acme_lookup_order with acme_get_order (facts only) + acme_search_knowledge (policy KB), forcing agents to use KB - Worker: recompute verdict after semantic judge, extract final_response from trajectory when engine field is empty - run_demo_audits: add --reuse-version flag, prefer newest set versions - devctl.sh: stable start/stop/status/restart for local dev stack Blocker: live LLM calls require a working endpoint. OpenRouter key is blocked by Cloudflare bot protection on POST requests; local Ollama (2b model) is too slow (~34min/token on CPU). Code changes are correct and verified via unit tests; E2E validation needs a fast LLM endpoint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Real-execution Agentic Audit demonstration on the Support Refund Assistant, stacked on PR #53.
What works (verified with live runs)
Live executions (Run #74, #75)
Known limitations (documented, not fabricated)
Changes
Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com