[FEATURE]: out-of-process host for existing Python CPEX plugins

[tedhabeck]

Summary

Closes: #20

Adds cpex-hosts-python, the crate that lets the Rust PluginManager run the
Python plugins that already exist — PII filters, identity resolvers, token
delegators — without rewriting them. It builds a per-plugin virtualenv, launches
the framework's worker.py in it as a long-lived subprocess, and speaks the same
newline-delimited-JSON stdio protocol the Python CLI already uses.

The point is migration continuity: a gateway can move to the Rust manager while
its existing Python plugins keep working unchanged.

Why

The Rust PluginManager cannot run a Python plugin on its own. This crate is the
Rust counterpart to the Python CLI's client.py + venv_comm.py. Rather than
port those as a one-to-one translation, it splits the three concerns — venv
build/cache, worker subprocess/protocol, and per-hook serialization — so each
tests in isolation and shared-package venv handling falls naturally to the venv
manager.

  PluginManager
    └── isolated_venv factory ──> IsolatedPythonPlugin ──┬── venv manager
                                   (initialize/shutdown) └── worker client
          └── hook adapter (serialize, send, deserialize)     │
                                                              ▼
                                        worker.py subprocess in the venv

Venv construction and worker launch happen in initialize() (rollback on
failure), teardown in shutdown(). Neither belongs on the invoke path — a cold
pip install is measured in minutes.

What's here

Venv lifecycle (venv.rs) — build, cache, and resolve per-plugin venvs,
including shared-package layouts. Cache metadata is keyed per host class.

Worker protocol (worker.rs) — subprocess supervision and the NDJSON stdio
framing, with the max_content_size frame bound enforced on both sides.

Hook dispatch (plugin.rs, conversion.rs, legacy/) — one adapter per
declared hook, carrying the native Pydantic payload shapes (ToolPreInvokePayload
and friends) rather than a generic wrapper, so existing plugins validate as-is.

Credentials (credentials.rs) — the framework strips raw tokens at every
process boundary (token fields are #[serde(skip)]), but identity and delegation
plugins genuinely need them. This adds a capability-gated wire DTO: a plugin
declaring read_inbound_credentials or read_delegated_tokens gets a dedicated
credential object built by reading the in-memory token directly. Production
credential types keep their serde guard and are never serialized. Fail-closed
rules and the residual exposure this does not close are documented in the
module.

Extensions delivery (extensions.rs) — the executor's capability-filtered
Extensions is serialized onto the task, so a 3-arg (payload, context, extensions) hook sees out-of-process what it would see in-process. Returns come
back through the executor's existing copy-on-write merge, which enforces the
mutability tiers; this host adds no tier logic of its own. Sensitive headers
(Authorization, Cookie, X-API-Key) are stripped in both directions,
case-insensitively, and raw_credentials never rides this channel.

Tests

172 passing (152 unit + 20 integration), 2 #[ignore]d by default because they
need an installed plugin with a built venv.

Suite	Covers
isolated_venv_e2e.rs	Adapter called directly against a scaffolded fixture
credential_e2e.rs	The capability-gated plaintext credential channel
extensions_merge_e2e.rs	Inbound wire shape; every mutability tier through the real executor
config_e2e.rs	YAML → factory → registry → invoke_by_name → executor → worker, against a plugin installed the way an operator installs one

extensions_merge_e2e.rs is worth a note on test design. The accept-side tier
outcomes can't be unit tested: http, security, and delegation writes are
gated by a WriteToken whose constructor is pub(crate) to cpex-core, so this
crate cannot mint one even in a test — which is exactly the property that makes
the gate trustworthy. Those paths are therefore driven through a real pipeline,
with tokens minted by the executor from declared capabilities. A fake_worker
stand-in covers the same merges without a Python subprocess by calling the
production parser on a canned response, so the code under test is identical and
only the subprocess is replaced.

Cross-surface verification, and one finding

The host and the Python worker land on different branches and ship separately, so
docs/specs/extensions-wire-contract.md pins the contract they share — neither
side's source can serve as the other's reference.

This branch ran the two against each other for the first time, via populated
Extensions on the tool_pre_invoke hook in config_e2e.rs. The channel works:
extensions cross into a real worker subprocess, are reconstructed there, and the
hook runs. It also overturned two claims the spec had recorded about the known
http-slot divergence, in the reassuring direction.

The divergence itself is unchanged and known — Rust HttpExtension has
request_headers/response_headers, Python has a single headers. The spec said
a Rust-shaped slot fails validation in the Python model and is dropped with a
warning. Both claims are wrong. Neither model sets extra="forbid" /
deny_unknown_fields, and both default their header maps, so unknown keys are
silently ignored and missing ones default to empty:

Direction	On the wire	Deserializes to	Warning
Rust → Python	{"request_headers": {"X-Request-Id": "r1"}}	HttpExtension(headers={})	none
Python → Rust	{"headers": {"X-Fine": "yes"}}	request_headers: {}, response_headers: {}	none

Verified empirically against both models, not inferred. The slot arrives
present-but-empty — the "hollow slot" this same contract explicitly rejects for
raw_credentials. A plugin reads extensions.http.headers, gets {}, and cannot
distinguish "no headers on this request" from "the headers didn't cross."

Two follow-on corrections went into the spec:

• reconstruct_extensions calls model_validate on the whole object, so a
  genuinely-failing slot would zero out all extensions, not degrade per-slot as
  the doc implied.
• PipelineResult::modified_extensions is the pipeline's final view and is
  always Some(...) on an allowed pipeline — not a "something changed" flag.
  Its own doc comment in executor.rs says otherwise; noted in the spec, comment
  left alone as out of scope here.

This is a correctness bug, not a security one. The sensitive-header strip runs
pre-serialization against the fields the sending model actually declares, so
Authorization never reaches the wire regardless of which shape the receiver
expects. config_e2e.rs asserts exactly that: no credential header value appears
anywhere in the serialized task JSON.

Known gaps

Tracked as Remaining work in the spec:

1. The http slot needs a mapping on one side. The other eleven slots cross
   correctly.
2. No test asserts a header value survives the round trip. This is the gap
   that let the divergence stay misdescribed: existing tests assert either on the
   wire JSON (Rust-shaped, so it passes) or on the hook running (it does). Write
   it as a failing test, then fix (1).
3. The extension models' permissive defaults are what turned a loud shape
   mismatch into a silent one. Tightening trades the deliberate slot-level
   version-skew tolerance, so it's a real design call rather than an obvious win.
4. The cross-surface run is not yet CI-reproducible, and this is the most
   important caveat on the verification above.

On (4), specifically

The worker side is unreleased (#113). Reproducing the run needs the plugin installed
and its venv's cpex replaced by hand:

cpex plugin --type test-pypi install "cpex-test-plugin@>=0.2.0"
plugins/<slug>/.venv/bin/pip install \
  "git+https://github.com/contextforge-org/cpex.git@feat/python_plugin_compat_0.1.x"
cargo test -p cpex-hosts-python --test config_e2e -- --ignored --nocapture

The version numbers collide. The branch build self-reports 0.1.2, and PyPI
publishes a cpex 0.1.2 that is a different artifact with no extensions support
(no EXTENSIONS_FIELD, no reconstruct_extensions). Nothing in pip list or the
dist-info version distinguishes them, so a venv can look correctly provisioned and
silently have no extensions channel. Check direct_url.json, not the version.

The consequence for reviewers: config_e2e.rs currently guards only on the venv
existing, so a registry-provisioned venv yields a green test that proves
nothing — the worker ignores the unknown extensions field, the hook still runs,
the marker still lands. The guard should grep the installed worker.py for
EXTENSIONS_FIELD, as testing::worker_delivers_extensions already does for the
CPEX_PYTHON_SOURCE path. Until then, treat a green config_e2e.rs as
conditional on having checked the venv's provenance.

Before merging

The working tree carries untracked artifacts that should be sorted out — none are
gitignored:

• docs/specs/extensions-wire-contract.md and the docs/plans/ +
  docs/brainstorms/ files — these are worth committing; the spec is
  referenced throughout this description.

Checks

• make lint passes
• make test passes
• CHANGELOG updated (if user-facing)