feat(plugins): capture harness events as producer envelopes#1064
feat(plugins): capture harness events as producer envelopes#1064phernandez wants to merge 8 commits into
Conversation
…velopes (#997) Signed-off-by: sourrrish <officialsourish0@gmail.com>
redact_payload() only inspected top-level string values, so a secret inside a nested dict or list passed through unredacted. Redaction now recurses over dicts, lists, and tuples; a denied key redacts its whole subtree rather than risking partial redaction inside it. Addresses maintainer review on #998. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: phernandez <paul@basicmachines.co>
append_to_event_log() wrote <cwd>/.basic-memory/events.jsonl into whatever repository the session ran in, dirtying user worktrees. The log now lives under the Basic Memory data dir at <data-dir>/events/<project-or-cwd-slug>/events.jsonl, resolved the same way as core's resolve_data_dir() (BASIC_MEMORY_CONFIG_DIR, then XDG_CONFIG_HOME, then ~/.basic-memory) without importing core. Rotation previously read the whole file on every hook write, which contradicted the documented append-only behavior. The hot path is now one append plus one stat; rotation only runs when the file passes cap x 512 bytes and drops the oldest half, so retention is documented as approximate. All open() calls pin encoding=utf-8 so envelopes read back identically across platform default encodings. DESIGN.md notes that a future version may write events through Basic Memory MCP/API calls instead of a local file (#997 leaves that open). Addresses maintainer review on #998. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: phernandez <paul@basicmachines.co>
The eventRetention setting was documented in settings.example.json and DESIGN.md but never read: every call site logged with the hard-coded default cap. All four hooks (Claude Code and Codex SessionStart and PreCompact) now pass the configured value through to append_to_event_log(), which validates it and falls back to the default for junk or non-positive values. Addresses maintainer review on #998. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: phernandez <paul@basicmachines.co>
Plugin marketplaces install a single plugin directory (plugins/ claude-code/ or plugins/codex/); nothing outside that tree ships. The hooks imported harness_envelope from plugins/shared/ two directories up, which only resolves in a repo checkout — on every installed plugin the import silently failed (_HAS_ENVELOPE=False) and users got no envelope capture at all. plugins/shared/ stays the canonical source. A new scripts/sync_plugin_shared.py vendors each shared module into every plugin's hooks/ directory as a committed copy, and the hooks now import from their own directory, which resolves identically in the repo checkout and the installed layout. Both plugin justfiles gain a vendor-check recipe (wired into check/ci-check, and therefore just package-check) that byte-compares the vendored copies against the canonical source so drift fails CI. Addresses maintainer review on #998. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: phernandez <paul@basicmachines.co>
Adds tests/test_harness_envelope.py for the shared module: recursive redaction over nested dicts and lists (including denied-key subtree redaction, nested path denial, truncation, and extra keys at depth), idempotency key stability within the minute window, event log location under BASIC_MEMORY_CONFIG_DIR / XDG_CONFIG_HOME with project-hint slugging, retention cap normalization, size-triggered rotation keeping the newest entries, a vendored-copy sync guard, and an installed-layout Codex run (bare plugins/codex copy, no plugins/shared sibling) proving the vendored import stamps envelopes. Extends the Claude hook harness with note-content capture, data-dir isolation, and a hooks_dir override, then covers: envelope provenance stamped on PreCompact checkpoints, opt-in event logging landing in the data dir (never the repo cwd), eventRetention reaching the rotation cap through the hook, and the installed plugin layout importing the vendored module. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: phernandez <paul@basicmachines.co>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6caeb79fc8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The Windows CI run exposed that path redaction never matched there:
os.path.expanduser("~/.ssh/") yields mixed separators on Windows
(C:\Users\x/.ssh/) while native payload values use backslashes, so
the startswith deny check was dead on that platform. Deny paths and
payload values now both normalize to forward slashes before
comparison, including user-supplied redactPaths. Adds a
platform-independent regression test exercising both separator
styles.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
captureEvents gated event capture with a truthiness check, so a hand-edited string value like "false" (truthy in Python) silently enabled recording. A privacy gate must fail closed: every gate site in the Claude Code and Codex SessionStart and PreCompact hooks now requires the JSON boolean true exactly. Tests cover string "true", string "false", and numeric values staying off in both plugins while checkpoint notes still write. Addresses Codex review on #1064. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: phernandez <paul@basicmachines.co>
|
Converting to draft: the producer-envelope direction stands, but we're not shipping the Claude hooks with this much Python embedded in shell heredocs. Before this lands, the hook logic moves into real Python files invoked by thin shell shims. Restructure plan to follow on this PR. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 736848fd35
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| log_path.parent.mkdir(parents=True, exist_ok=True) | ||
|
|
||
| # Append the new event | ||
| line = envelope_to_json(envelope) + "\n" | ||
| with open(log_path, "a", encoding="utf-8") as fh: |
There was a problem hiding this comment.
When captureEvents is enabled and this canonical module (byte-copied into both plugin hook dirs) creates a fresh Basic Memory config/events tree on a normal 022 umask, mkdir() leaves the directories world-searchable and open(..., "a") creates events.jsonl as 0644; I verified this path with a fresh BASIC_MEMORY_CONFIG_DIR. That makes session ids, cwd/project hints, and payload summaries readable by other local users, so the privacy-gated event log should create/chmod the events directories to 0700 and the JSONL file to 0600 like the core config path does.
Useful? React with 👍 / 👎.
|
Closing in favor of the CLI-in-core direction decided on #997 (SPEC-55 revision 2026-07-15): hook logic moves into |
2 participants
What this does
Adds an opt-in "harness WAL" producer layer for the Claude Code and Codex plugins (issue #997, SPEC-55). Supported hook events (
SessionStart,PreCompact) are normalized into a sharedHarnessEnvelopeshape — source, session id, turn id, cwd, project hint, hook name, timestamp, redacted payload summary, and a deterministic idempotency key.envelope_source,envelope_event,envelope_hook,idempotency_key) and[source]/[hook]/[event]/[idempotency]provenance observations, making checkpoints queryable by producer and dedup-safe.captureEvents: true(off by default) additionally appends raw envelopes to a local JSONL event log for later coalescing by SPEC-61 memory routines.ToolLedgerpicoschemas for whenPostToolUse-style hooks land (v1).Carried forward from #998
The envelope design and base implementation are the work of @sourrris — their commit from #998 is cherry-picked onto this branch with authorship intact (
feat(plugins): capture Claude and Codex harness events as producer envelopes (#997)). TheHarnessEnvelopedataclass, idempotency scheme, layered redaction strategy, provenance/frontmatter helpers, DESIGN.md section 14, and the ToolLedger schemas are all theirs. The Claude Code hook integration was re-applied by hand to fit the settings-resolution rework that landed on main after #998 branched; the design is unchanged.This branch exists because the remaining fixes touched packaging and repo conventions that were easier to land from a repo-owned branch. Thank you @sourrris for a solid v0.
Hardening added on top
plugins/shared/import silently failed (_HAS_ENVELOPE=False) for every installed user.scripts/sync_plugin_shared.pynow vendors the canonical module into each plugin'shooks/dir (committed copies, byte-compared by a newvendor-checkrecipe wired intojust package-check), and hooks import from their own directory.redact_payload()only redacted top-level strings; nested dicts/lists now redact at any depth, and a denied key redacts its whole subtree.events.jsonlno longer lands in<cwd>/.basic-memory/; it lives at<data-dir>/events/<project-or-cwd-slug>/events.jsonl, resolving the data dir like core'sresolve_data_dir()(BASIC_MEMORY_CONFIG_DIR→XDG_CONFIG_HOME→~/.basic-memory). DESIGN.md notes a future version may write through MCP/API instead (Capture Claude/Codex harness events as Basic Memory producer envelopes #997 leaves that open).eventRetentionwired through — the documented setting was dead; all four hooks now pass it to the rotation cap (validated, with fallback to the default).stat(); rotation only triggers past a size threshold and retention is documented as approximate.encoding="utf-8"pinned on all event logopen()calls.Test plan
tests/test_harness_envelope.py(new, 16 tests): recursive redaction (nested dicts, lists, denied-key subtrees, path denial and truncation at depth, extra keys), idempotency key stability/divergence, event log location honoringBASIC_MEMORY_CONFIG_DIR/XDG_CONFIG_HOMEwith project-hint slugging, retention cap normalization, size-triggered rotation keeping newest entries, vendored-copy sync guard, and an installed-layout Codex run (bareplugins/codexcopy with noplugins/sharedsibling) proving the vendored import stamps envelopes end to end.tests/test_claude_plugin_hooks.py(extended, 4 new tests): envelope frontmatter + provenance observations on real PreCompact checkpoint writes, event logging opt-in and located in the data dir (never the repo cwd),eventRetentionreaching rotation through the hook, and an installed-layout Claude Code run via a bare plugin copy.tests/test_codex_plugin_package.py(5) passes.just package-checkpasses (includes the newvendor-checkin both plugin checks plusclaude plugin validate . --strict).just typecheckpasses;ruff check/ruff formatclean on touched files.Fixes #997
Supersedes #998
🤖 Generated with Claude Code