Skip to content

Open the source dimension + stable envelope ingest for third-party agents - #56

Merged
TensorTemplar merged 1 commit into
mainfrom
feat/open-source-ingest
Aug 19, 2026
Merged

TensorTemplar merged 1 commit into
mainfrom
feat/open-source-ingest

Conversation

@TensorTemplar

Copy link
Copy Markdown
Owner

Addresses #46 (ingestion interface for non-Claude-Code agent traces). Builds directly on the protocol from #54 — same taxonomy, same adapter/dispatch architecture — and adds exactly what #46's discussion still asked for after #54 landed.

What was still missing after #54

@Necmttn's final review of #54 (in #46) flagged the remaining gap: AbstractEventSource was still a closed claude_code | opencode enum, so third-party collectors could not plug in without patching core. And @botbotfromuk's original question — watched file vs explicit import call — had no answer yet.

Changes

1. Open source dimension

  • AbstractHookEvent.source is now an open string; AbstractEventSource is documented vocabulary ("this source has a wire-format adapter"), not a gate.
  • The DB reader no longer casts stored sources through the enum — third-party source names round-trip storage intact (covered by a round-trip test).
  • emit-event remains adapter-only by design; its help now points adapter-less collectors at ingest.

2. Stable external envelope — core/protocol/schema.py
EventEnvelope, per the shape proposed in #46:

{"source": "mmkr", "schema_version": 1, "session_id": "...", "event_id": "...",
 "parent_session_id": null, "seq": 42, "occurred_at": "...", "kind": "tool_call_started",
 "tool_name": "...", "duration_ms": 88, "exit_code": 0, "error_message": null, "raw": {}}
  • kind is the closed AbstractEventType taxonomy from Extract common hook protocol + experimental memory extraction #54 — validated, no overloading of notification required since ticks map onto real lifecycle kinds.
  • event_id is required so backfills are always idempotent; schema_version fails fast on anything but 1.
  • Unknown fields are preserved (extra="allow") for forward compatibility.

3. slopometry ingest --source <name> [--file] [--working-directory]
The explicit import call answering #46: JSONL envelopes from stdin or file, atomic validate-then-insert (a bad line never causes a partial ingest), loud exit-2 failures for invalid envelopes or source/--source mismatch. Dedupe on (source, event_id) — re-ingesting a trace reports skipped N duplicates. Sequence numbers continue after the highest stored/batched seq when envelopes omit seq. Envelope ingest deliberately does not capture git/project session context (the ingester's repo is unrelated to the traced session).

4. Migration020

  • Adds event_id + unique partial index on (source, event_id) WHERE event_id IS NOT NULL.
  • Repairs event_type values written by the abandoned feat/hook-protocol branch (1:1 remap tool_call→tool_call_started, stop→turn_completed, etc.); databases that never ran that branch are unaffected.
  • Numbered 020 because 017–019 were consumed by diverged local trees.

For reviewers from #46

@botbotfromuk — your SlopometryCollector can now emit envelope JSONL instead of Claude hook names: convert_trace_to_slopometry() backfills are safe to re-run any number of times thanks to (source, event_id). tick_start/decision/action map onto notification/turn_completed as fits; keep finer distinctions in raw until real consumers need first-class kinds.

@Necmttn — the closed-enum caveat from your review is resolved here without touching the taxonomy: source is open, AbstractEventSource is constants-only, nothing gates on membership.

Testing

  • New: test_protocol_schema.py (envelope validation), test_protocol_ingest.py (canonical storage, timing fields, open-source round-trip, seq assignment/interleaving/continuation, idempotent re-ingest, CLI incl. --file and all failure paths), migration-020 test incl. index integrity.
  • Full suite: 1054 passed standalone (2 pre-existing env failures); ruff clean; mypy 39 vs 40 on main — no new errors.
  • Live-verified: installed per README, migration 020 applied to a 900k-event production DB (dialect repair + index), ingest + re-ingest + solo ls all correct.

Note: #55 was a previous, larger attempt at this that unknowingly re-implemented #54 with a divergent taxonomy; it was closed in favor of this focused delta.

…ents

Builds on the #54 protocol: keeps its AbstractEventType taxonomy and
adapter/dispatch architecture, and adds what #46 still asked for.

- AbstractHookEvent.source is now an open string; AbstractEventSource
  becomes documented vocabulary ("has an adapter"), not a gate. The
  DB reader no longer casts sources through the enum, so third-party
  source names round-trip storage intact.
- EventEnvelope (protocol/schema.py): the stable external contract —
  source (open), schema_version (fail-fast), required event_id,
  canonical kind, seq, occurred_at, tool fields, raw payload.
- `slopometry ingest --source <name> [--file] [--working-directory]`:
  JSONL envelopes from stdin/file, atomic validate-then-insert, loud
  exit-2 failures. Dedupe on (source, event_id) makes trace backfills
  idempotent; sequence numbers continue after the highest stored/batched
  seq when envelopes omit one.
- Migration020: event_id column + unique partial index (source, event_id)
  WHERE event_id IS NOT NULL; also repairs event_type values written by
  the abandoned hook-protocol branch (1:1 remap back onto the canonical
  taxonomy). Numbered 020 because 017-019 were consumed by diverged trees.
@github-actions

github-actions Bot commented Aug 18, 2026 •

Copy link
Copy Markdown

📊 Slopometry QPE Report

QPE Score: 0.2524

Metric Value Description
MI (normalized) 0.293 Maintainability Index / 100
Smell Penalty 0.207 Weighted code smell deduction
Adjusted Quality 0.252 MI × (1 - smell_penalty) + bonuses
Code Smell Breakdown
Smell Count
inline import 256
orphan comment 214
dict get with default 93
swallowed exception 33
single method class 29
passthrough wrapper 17
hasattr getattr 15
untracked todo 8
test skip 6
type ignore 3
acknowledged silent except 1
nonempty init 1

Higher QPE = better quality

@github-actions

github-actions Bot commented Aug 18, 2026 •

Copy link
Copy Markdown

📈 Slopometry Impact Report

Impact: NEUTRAL (score: 0.244)

Metric Delta Description
QPE +0.0048 Quality-Per-Effort change
MI +184.136 Maintainability Index change
CC +128.000 Cyclomatic Complexity change
Effort +678667.4 Halstead Effort change
Count
Changed files 11
Blind spots 4
Smell Advantage Breakdown
Smell Baseline Current Weighted Delta
type ignore 4 3 -0.0800
dict get with default 92 93 +0.0500
single method class 28 29 +0.0500
orphan comment 210 214 +0.0400
inline import 252 256 +0.0400

Source: Previous commit (48992c2 vs 1b516ec)

@TensorTemplar
TensorTemplar force-pushed the feat/open-source-ingest branch from 022e383 to 3811d80 Compare August 19, 2026 08:24
@TensorTemplar
TensorTemplar merged commit e343f7a into main Aug 19, 2026
4 checks passed
@TensorTemplar
TensorTemplar deleted the feat/open-source-ingest branch August 19, 2026 08:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant