Open the source dimension + stable envelope ingest for third-party agents - #56
Merged
Merged
Conversation
…ents Builds on the #54 protocol: keeps its AbstractEventType taxonomy and adapter/dispatch architecture, and adds what #46 still asked for. - AbstractHookEvent.source is now an open string; AbstractEventSource becomes documented vocabulary ("has an adapter"), not a gate. The DB reader no longer casts sources through the enum, so third-party source names round-trip storage intact. - EventEnvelope (protocol/schema.py): the stable external contract — source (open), schema_version (fail-fast), required event_id, canonical kind, seq, occurred_at, tool fields, raw payload. - `slopometry ingest --source <name> [--file] [--working-directory]`: JSONL envelopes from stdin/file, atomic validate-then-insert, loud exit-2 failures. Dedupe on (source, event_id) makes trace backfills idempotent; sequence numbers continue after the highest stored/batched seq when envelopes omit one. - Migration020: event_id column + unique partial index (source, event_id) WHERE event_id IS NOT NULL; also repairs event_type values written by the abandoned hook-protocol branch (1:1 remap back onto the canonical taxonomy). Numbered 020 because 017-019 were consumed by diverged trees.
📊 Slopometry QPE ReportQPE Score: 0.2524
Code Smell Breakdown
|
📈 Slopometry Impact ReportImpact: NEUTRAL (score: 0.244)
Smell Advantage Breakdown
|
TensorTemplar
force-pushed
the
feat/open-source-ingest
branch
from
August 19, 2026 08:24
022e383 to
3811d80
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #46 (ingestion interface for non-Claude-Code agent traces). Builds directly on the protocol from #54 — same taxonomy, same adapter/dispatch architecture — and adds exactly what #46's discussion still asked for after #54 landed.
What was still missing after #54
@Necmttn's final review of #54 (in #46) flagged the remaining gap:
AbstractEventSourcewas still a closedclaude_code | opencodeenum, so third-party collectors could not plug in without patching core. And @botbotfromuk's original question — watched file vs explicit import call — had no answer yet.Changes
1. Open source dimension
AbstractHookEvent.sourceis now an open string;AbstractEventSourceis documented vocabulary ("this source has a wire-format adapter"), not a gate.emit-eventremains adapter-only by design; its help now points adapter-less collectors atingest.2. Stable external envelope —
core/protocol/schema.pyEventEnvelope, per the shape proposed in #46:{"source": "mmkr", "schema_version": 1, "session_id": "...", "event_id": "...", "parent_session_id": null, "seq": 42, "occurred_at": "...", "kind": "tool_call_started", "tool_name": "...", "duration_ms": 88, "exit_code": 0, "error_message": null, "raw": {}}kindis the closedAbstractEventTypetaxonomy from Extract common hook protocol + experimental memory extraction #54 — validated, no overloading ofnotificationrequired since ticks map onto real lifecycle kinds.event_idis required so backfills are always idempotent;schema_versionfails fast on anything but 1.extra="allow") for forward compatibility.3.
slopometry ingest --source <name> [--file] [--working-directory]The explicit import call answering #46: JSONL envelopes from stdin or file, atomic validate-then-insert (a bad line never causes a partial ingest), loud exit-2 failures for invalid envelopes or source/
--sourcemismatch. Dedupe on(source, event_id)— re-ingesting a trace reportsskipped N duplicates. Sequence numbers continue after the highest stored/batched seq when envelopes omitseq. Envelope ingest deliberately does not capture git/project session context (the ingester's repo is unrelated to the traced session).4. Migration020
event_id+ unique partial index on(source, event_id) WHERE event_id IS NOT NULL.event_typevalues written by the abandonedfeat/hook-protocolbranch (1:1 remaptool_call→tool_call_started,stop→turn_completed, etc.); databases that never ran that branch are unaffected.For reviewers from #46
@botbotfromuk — your
SlopometryCollectorcan now emit envelope JSONL instead of Claude hook names:convert_trace_to_slopometry()backfills are safe to re-run any number of times thanks to(source, event_id).tick_start/decision/actionmap ontonotification/turn_completedas fits; keep finer distinctions inrawuntil real consumers need first-class kinds.@Necmttn — the closed-enum caveat from your review is resolved here without touching the taxonomy: source is open,
AbstractEventSourceis constants-only, nothing gates on membership.Testing
test_protocol_schema.py(envelope validation),test_protocol_ingest.py(canonical storage, timing fields, open-source round-trip, seq assignment/interleaving/continuation, idempotent re-ingest, CLI incl.--fileand all failure paths), migration-020 test incl. index integrity.solo lsall correct.Note: #55 was a previous, larger attempt at this that unknowingly re-implemented #54 with a divergent taxonomy; it was closed in favor of this focused delta.