Repository navigation
Refuse sessions with non-object event data instead of printing a traceback - #293
Merged
Siddhant-K-code merged 5 commits intoOct 25, 2026
Merged
Conversation
Commands that read keys from event data crashed with a Python traceback when `agent-strace server` stored array, string, number, or boolean data, or a hand edit stored null. They now refuse the session the way `share` does since #287: one `Error:` line on stderr, exit code 1, and nothing printed, written, sent, or opened. - Move NonObjectEventDataError and require_object_event_data to event_data.py and keep the disclosure import path. The helper can limit the check to the event types or positions a command reads, and can name the session. - Catch NonObjectEventDataError once in the CLI handler wrapper. Telemetry records exit code 1 with no error_type, as it does for share. - Each command checks only the events whose data it reads, so a session keeps working with commands that never read the bad event. - watch skips its checks for a live event with non-object data and prints one warning. watch --compaction-checkpoint and mcp-scan --watch refuse at startup, and mcp-scan --watch skips live events with a warning. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
export --format otlp-genai refused a session when a tool_call with non-object data was paired with an error event, although session_to_otlp_genai never reads that tool_call's data. Replace OTLP_GENAI_DATA_EVENT_TYPES with otlp_genai_data_indexes, which pairs llm_request and tool_call events by ID the way the converter does. export --format temporal printed a traceback, with -o and with --endpoint, when the session had a Temporal trace context and non-object data. session_to_temporal_otlp now runs the same check right after it validates the context, so a session without one keeps its "Temporal export failed" message. cmd_export and export_temporal_otlp re-raise NonObjectEventDataError before their ValueError handlers so the standard Error line prints. Add tests/test_non_object_event_data_mirrors.py. It stores non-object data in one event at a time and runs each command whose check mirrors its reads twice: as shipped, and with require_object_event_data patched to a no-op. The shipped run must refuse exactly when the unchecked run raises. Removing any of five checks that no earlier test covered now fails it: the mcp-scan --session check on an earlier session_start, the optimize LLM-path check, the sample --deduplicate tool_call check, and both postmortem_data_indexes scans. Fix the postmortem docstring that pointed at tests that did not exist, and document the temporal and optimize behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
session_to_otlp keeps one pending tool_call per event ID. When a second tool_call reuses the ID of a tool_call that is still pending, the converter drops the first one and never reads its data. The check read every tool_call, so export --format otlp refused such a session, which main exports with exit code 0. A server client that posts its own event IDs can store such a session. otlp_data_indexes now pairs tool_call events by ID the way the converter does. This is separate from the review fixes so it can be dropped on its own. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
In the export fixture, a tool_call now reuses the ID of the tool_call that an error paired with earlier. Plain otlp reads that earlier tool_call when the error pairs with it, so otlp_data_indexes must pair errors as well as tool_results. Before this, changing (EventType.TOOL_RESULT, EventType.ERROR) to (EventType.TOOL_RESULT,) passed the full suite while export --format otlp printed a traceback. The reused ID alone would hide a missing pop in the otlp-genai error branch, so the fixture also gets a tool_call that an error pairs with and whose ID nothing reuses. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Three more single-line changes to the pairing helpers passed the full suite while changing what the exports do: - otlp_data_indexes and otlp_genai_data_indexes not pairing a tool_result with its tool_call. A later tool_call with the same ID then hides the earlier one, which the converter reads, so export printed a traceback. - otlp_genai_data_indexes pairing an llm_response that names its request with the oldest open request instead. That refuses sessions that otlp-genai exports. The export fixture now reuses the ID of a tool_call after a tool_result pairs with it, and pairs an llm_response by parent while an older request is open and later replaced. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Siddhant-K-code
deleted the
siddhant-k-code-non-object-event-data-tracebacks
branch
October 25, 2026 14:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Commands that read keys from event
dataprinted a Python traceback when an event'sdatawas not a JSON object.agent-strace serverstores array, string, number and booleandatafrom POST /events, and a hand edit can store null, so these sessions exist in normal use. With this change, those commands refuse the session the same waysharehas since #287: they write oneError:line to stderr, exit with code 1, and print, write, send or open nothing. The one exception isoptimizewith an LLM configured, described in decision 1.On 024814f,
agent-strace explainon such a session ends with a full traceback andAttributeError: 'str' object has no attribute 'get'. With this branch:Both exit with code 1. There is no version bump and no release file change.
Changes after review at e3dc9a3
Both new commits change only
tests/test_non_object_event_data_mirrors.py.src/is the same as at e3dc9a3, so the case table, the sweep and the other results below still describe this branch.otlp_data_indexes, changingevent.event_type in (EventType.TOOL_RESULT, EventType.ERROR)to(EventType.TOOL_RESULT,)passed the full suite, whileexport --format otlpprinted a traceback.42, an error that pairs with it, a tool_call that reuses its ID and a tool_result that pairs with that.export --format otlp -o out.jsonon main and on the mutant printsAttributeError: 'int' object has no attribute 'get'. This branch printsError: session event data must be an object: event 2 of 6 (tool_call) has number data.export otlp -oandexport otlp --endpoint, withtool_call e1e2e3e4e5e60011 #15 = 42: raised AttributeError: 'int' object has no attribute 'get' (otlp.py:316). The full suite fails with that one failure.otlp_data_indexesorotlp_genai_data_indexesnot pairing a tool_result with its tool_call. Event 35 reuses the ID of the tool_call at event 13, after the tool_result at event 14 pairs with it. Take a session with a tool_call with data42, a tool_result that pairs with it, and a tool_call that reuses its ID. Withexport --format otlp -oand--format otlp-genai -o, main prints a traceback, each mutant prints one for its format, and this branch printsError: session event data must be an object: event 2 of 5 (tool_call) has number data.otlp_genai_data_indexespairing an llm_response that names its request with the oldest open request instead. Events 36 to 38 pair an llm_response with its request by parent while an older request is open, and then replace the older request. On a session shaped like that, with data42on the older request, main and this branch export it with--format otlp-genai -oand exit 0. The mutant refuses it withevent 2 of 6 (llm_request) has number data.Changes after review at 8929bcc
export --format otlp-genaino longer refuses a session because of a tool_call that an error event pairs with.session_to_otlp_genainever reads that tool_call, and main exports the session with exit code 0. It also no longer refuses because of an llm_request or tool_call that a later event with the same ID replaced before anything paired with it.otlp_genai_data_indexesreplacesOTLP_GENAI_DATA_EVENT_TYPES. See decision 4.export --format temporalno longer prints a traceback, with-oor--endpoint, on a session that has a Temporal trace context and non-object data. See decision 11.tests/test_non_object_event_data_mirrors.py, described under Tests. Removing any of the five checks that the review found untested now fails it.postmortem_data_indexesdocstring now names the test that checks it, anddocs/commands.mddescribes the temporal and optimize behavior.export --format otlpno longer refuses a session because of a tool_call that a later tool_call with the same ID replaced before anything paired with it. See decision 12. At that point you could drop it on its own. 96fc036 and a140125 now build on its test rows.The same cases on main, at 8929bcc and on this branch, each run in a temporary cwd and HOME:
export --format temporal,-oor--endpoint, Temporal context, tool_result with array dataError: session event data must be an object: event 4 of 5 (tool_result) has array data, exit 1export --format otlp-genai -o, tool_call with data42that an error pairs withexport --format otlp-genai --endpointto a closed port, same sessionOTLP GenAI export failed, exit 1export --format otlp -o, tool_call with data42replaced by a tool_call with the same IDWhat changed
src/agent_trace/event_data.py. It holds:NonObjectEventDataError, aValueError.require_object_event_data(events, *, event_types=None, indexes=None, session_id=None). The message gives the 1-based position, the event type and the JSON type, and counts every other non-object event in the session. It never includes data values or event IDs.skipped_event_warning, used by the live watchers.disclosure.pyre-exports both names, sofrom agent_trace.disclosure import NonObjectEventDataError, require_object_event_datastill works.shareoutput does not change.cli.py:_run_handler, called by_run_with_product_telemetry, writesError: <message>to stderr and returns 1.cli.pyalso checks the data forstats,export --format otlpandexport --format otlp-genai. For the two export formats it checks the events thatotlp_data_indexesandotlp_genai_data_indexesinotlp.pyreturn. It re-raisesNonObjectEventDataErrorbefore itsexcept ValueError, so a temporal refusal prints the standard line.require_object_event_data: a2a, audit, budget_report, cognitive_debt, compaction, compare, compliance, context_score, cost, dashboard, diff, explain, inflation, langfuse_export, mcp_scan, oncall, optimize, policy, postmortem, pr_comment, replay, sample, standup, subagent, team_report, temporal, tenancy, timeline, token_budget and why.otlpholds the index helpers for the export checks, andwatchusesskipped_event_warning.temporal.pyruns the otlp-genai check insession_to_temporal_otlp, right after_validate_ids.export_temporal_otlpre-raisesNonObjectEventDataErrorbefore itsexcept ValueError.docs/commands.mdhas a new section, "Sessions with non-object event data". It lists the refusing commands, the message format, the commands that keep working, and the watch and mcp-scan behavior. Thesharesection links to it. It also says which export formats pair events by ID, that temporal checks the Temporal trace context first, and whenoptimizewith an LLM can refuse after it sent the request.tests/test_event_data.py,tests/test_non_object_event_data.pyandtests/test_non_object_event_data_mirrors.py, described below.Decisions
share.optimizewith--base-urland--api-keyis the one exception. It checks the events in its prompt before it sends the LLM request, so a refusal there sends nothing. When the LLM returns no patterns, it falls back to its heuristics, which read more events, so it can refuse after it sent the request. Main sends the same request and then crashes.NonObjectEventDataErrorfrom the shared helper, and_run_handlerprints it. No command formats its own error line.cost --breakdown,cost --tenantandtenant reportalready catchValueErrorand printError: {exc}. That is the same line, so I left those handlers alone.event_data.pyand keep the old import path. About 30 commands with nothing to do with share disclosure now use the check, so it lives in its own module next to the watch warning helper.disclosure.pyre-exports both names.verify,listandexport(json, ndjson, csv) unchanged.explainstill works when only a file_read event is bad, andreplay --limit 5still works when the bad event comes after the fifth event. The tests compare such cases against a clean store.otlp_genai_data_indexesandotlp_data_indexesreplay each converter's pairing and return the events whose data it reads:otlp-genaiandtemporalread an llm_request when an llm_response pairs with it or when it is still unpaired at the end, and a tool_call when a tool_result pairs with it or when it is still unpaired at the end. They do not read a tool_call that an error pairs with, or an llm_request or tool_call that a later event with the same ID replaced while it was unpaired.otlpreads every llm_request. It reads a tool_call when a tool_result or an error pairs with it or when it is still unpaired at the end, but not when a later tool_call with the same ID replaced it while it was unpaired.replayrefuses up front instead of printing bad events raw.replay --format htmlandreplay --diffwrite files, which must not happen for a session they cannot read.agent-strace export --format jsonstill shows the raw data.replay --format htmlcopies data withdict(), so it refuses only data thatdict()rejects. An empty array, an empty string or an array of key-value pairs still renders as before.watchkeeps running and skips the checks for the bad event.--stream-to. When a check would read its data, watch skips the checks for that event and writes one line to stderr, for example[watch] Skipped checks for one tool_call event because its data is an array, not an object.watch --compaction-checkpointrefuses at startup when the stored session already has non-object data, because the checkpoint watcher reads every stored event when it starts. Live events with non-object data are left out of checkpoints.mcp-scan --watchrefuses at startup when the session history has non-object data, becausescan_behavioural_anomaliesreads every history event. Live events with non-object data are skipped with one[mcp-scan]warning line, and a non-object session_end still ends the watch.pr-comment,policy generateandmcp-scan --sessionname it even for one session, because they share that code with their multi-session modes.share.error_type. The catch point returns 1 instead of raising, so the completion event hassuccess: falseandexit_code: 1with noerror_type, the same as a session refused byshare.export --format temporalchecks the Temporal trace context first.session_to_temporal_otlpruns the otlp-genai check right after_validate_ids, so a session without a usable context keeps itsTemporal export failedmessage and exit code 1.NonObjectEventDataErroris aValueError, socmd_exportandexport_temporal_otlpre-raise it before theirexcept ValueErrorhandlers. Without that, the refusal would print asTemporal export failed: session event data must be an objectinstead of the standardError:line.Command sweep, before and after
I built a fixture store with three sessions:
I ran 184 command variants, covering every command and mode that reads session events, on base 024814f and on this branch. Each run used a temporary cwd and HOME, refused all network connections and stubbed the browser.
[], an array of key-value pairs, an array of key names,0,false,""and a string of key names. These values slip pastdata or {},dict(data)andinchecks.verify --from-export exp.json.exp.jsonwithexport --format eu-ai-act --output exp.json. That export crashed on base and refuses now, so the file never exists andverifyraisesFileNotFoundErroron both.verify --from-export missing.jsonraises the same error on base with an empty store. It never reads event data, so this PR leaves it alone.watch: 8 variants over 81 scenarios, 648 runs.--compaction-checkpoint).agent-strace mcpandserver: the stdio tools (81 scenarios) and theserverGET routes give the same responses as on base.cmd_export,otlp.py,temporal.pyand a postmortem docstring. I re-ran the 16exportvariants on fresh fixtures, 3088 runs each on base, 8929bcc and e3dc9a3. Their counts are the same as in the sweep at 8929bcc. Every e3dc9a3 run matches 8929bcc byte for byte, except 26eu-ai-actruns that print the current time and match after normalizing.Commands that printed a traceback on base and now refuse (84 sweep variants, plus
export --format temporal):a2a-treeaudit(also with--verify-chainand--policy)budget-reportcognitive-debtcompaction(also with--diffand--behavior-diff)compare(decision events,--last,--rerun)compliance exportcontext-score --filecost(session,--since,--csv,--breakdown,--tenant)dashboard --trenddiff(also with--compareand--semantic)explainexport --format otlp,otlp-genai,temporalandeu-ai-act.temporalcrashes only on a session with a Temporal trace context. The sweep fixture had none, so the review at 8929bcc found it, and the sweep counts above leave it out.export --backend otlpandlangfuseinflationmcp-scan --sessiononcalloptimize --dry-runpolicy generatepostmortem(session and--list)pr-comment --dry-runreplay(text,--filter,--limit,--expand-subagents,--diff,--format html)sample(every strategy)standupstatsteam-reporttenant reporttimelinetoken-budgetwhyThe earlier sweep listed
cognitive-debt,context-score,inflationandtoken-budgetas safe. They crash only when the bad event has a type they read:cognitive-debton tool_call or file_write;context-score --fileon tool_call;inflationon user_prompt, llm_request, llm_response or assistant_response;token-budgeton llm_request or llm_response.These commands exit 0 in every scenario before and after (68 variants):
annotate,audit-readiness,audit-toolsbackfill-meta,baseline,config-watch affected,retention,tenant deleteandtenant exportcontext-score,curve,dashboard,drift --save-baseline,fingerprint,freshness,healtheval compare,eval datasetexport --format json,ndjsonandcsv,identity signandidentity verifyinspect,lint,list,replay --tree,stats --include-subagents,tree,verify, andwhyon the first eventmcp,mcp-scanwithout--sessionpolicy backtest,policy coverage,policy diffskills,skills-evalfreeze,regression,compareon non-decision events,compliance-report,org-reportandshare --assignmentalready exited 1 with their own message, and they still do.Tests
tests/test_event_data.py(11 tests) covers:event_typesandindexesfilters;disclosureimport path.tests/test_non_object_event_data.py(21 tests) ports the sweep fixture. It runsmain()in-process with network connections refused, the browser stubbed and telemetry captured.success: false,exit_code: 1and noerror_type. It also asserts no network, no browser and no file changes.explain,timeline,cost,statsandreplayrefuse each of the five JSON types.explain,timelineandcostand asserts that stderr has no traceback.verifystill checks the chain,exportjson and csv keep the raw data, andlistshows every session.replay --format htmlstill renders data thatdict()accepts.--compaction-checkpointrefuses at startup.tests/test_non_object_event_data_mirrors.py(8 tests) checks the commands whose check mirrors what they read: the OTLP and Langfuse exports, temporal, replay, why, token-budget,compare --rerun, postmortem, sample,mcp-scan --session,replay --expand-subagentsand optimize.require_object_event_datapatched to a no-op in every agent_trace module that imports it.sample --deduplicate;mcp-scan --session.optimizeruns with--base-url,--api-keyand a patched_call_llm. When the prompt's events refuse, the LLM is not called. When the LLM returns no patterns, the one heuristics refusal comes after one call.The extra skip on Python 3.10 is the existing tomllib test that needs Python 3.11.
The new tests fail without the fix. Against
git archive 024814f src:tests/test_non_object_event_data.pyreportsFAILED (failures=4, errors=151), with subtests counted one by one. 13 of its 21 tests fail. The 8 that pass check behavior this PR keeps: the 2 fixture tests, the 5 unchanged-behavior tests, and the watch test for events that no check reads.tests/test_event_data.pyandtests/test_non_object_event_data_mirrors.pycannot importagent_trace.event_data.tests/test_non_object_event_data_mirrors.pyalso fails on this branch when any one of these is undone, one at a time:mcp-scan --sessioncheck of an earlier session's session_start, the optimize LLM-path check, thesample --deduplicatecheck, and bothpostmortem_data_indexesscans;otlp_data_indexes, the mutant from the review at e3dc9a3 (caught since 96fc036);otlp_data_indexesand inotlp_genai_data_indexes, and the parent match for an llm_response inotlp_genai_data_indexes(caught since a140125).I ran 46 mutants against each committed version of this test file. 45 are small changes to the checks, the re-raises and the pairing helpers
otlp_data_indexes,otlp_genai_data_indexesandlangfuse_data_indexes. The last one puts back 8929bcc'sotlp.py,cli.pyandtemporal.py.The 3 that survive read a tool_call or request when it pairs but leave it in the pending map, one in each pairing helper. They return the same indexes as the shipped code, because a tool_call or request they read again was already read when it first paired, so no test can tell them apart.
With 8929bcc's
otlp.py,cli.pyandtemporal.py, the same test reports otlp-genai refusing at events 7, 9, 12, 15, 20, 29 and 32, which it never reads, plain otlp refusing at event 29, and temporal raising at 27 events.Not changed
These items are out of scope:
argumentsis a list or whosetool_nameis not a string.explainand plainsharestill print a traceback for that, andshare --dry-runstill exits 0.dataon POST /events. Rejecting it at ingestion is a separate change.Existing behavior this PR leaves as it is:
verify --from-exporton a missing file still raisesFileNotFoundError. It never reads event data.eval dataset autowithout--datasetstill raises its existingIsADirectoryError. It does the same on base with a clean store.freeze,regression,compareon non-decision events,compliance-report,org-reportandshare --assignmentkeep their existing message and exit code 1.lintandeval gatestill print one[lint] Rule '<name>' failed: <error>warning for each affected rule and keep their exit code.agent-strace mcptools keep their JSON-RPC -32603 error text.serverroutes are not affected.mcp-scanwithout--sessionstill skips sessions it cannot read without a message.watch --stream-towith--stream-otlpstill drops a batch that holds non-object data without a message, as on base.Exception.