Route PEP 723 scripts to inline environments (PEP 723 PR 9/16)

[Stella Huang (StellaHuang95)]

Roadmap context

This is PR 9 of 16 in the PEP 723 inline-script roadmap and the repository's intentional routing cutover. Earlier manager PRs create, discover, validate, and persist environments; this PR makes normal per-file environment lookup use them when a saved script association is proven current.

Phase 2/3	PR	Status
	PR 7: per-script persistence	merged (#1697)
	PR 8: activation-time cache discovery	merged (#1722)
	PR 9: automatic per-script routing	this PR
	PR 10: exact script project registration	follow-up
	PRs 11-12: CodeLens and bulk setup UX	follow-up
	PRs 17-19: Pylance/Python debugger integration	cross-repository follow-up

Why this PR

The inline manager can already create/reuse environments, persist a script association, rediscover cache entries after restart, clear state safely, and report lifecycle telemetry. Nothing automatically selects that manager for a Python file, however. A script continues to use its normal project/default environment unless another component directly asks the inline manager.

Automatic routing must be stricter than checking whether a file contains PEP 723 metadata or whether Memento contains an executable path. It must prove both:

current saved PEP 723 metadata exists
                  +
a persisted environment association is valid for that metadata and cache generation
                  =
route this file to InlineScriptEnvManager

If either proof is absent, dirty, stale, transiently unavailable, or superseded by an explicit choice, routing falls through to the existing project/default manager.

What this PR does

Adds an activation-scoped routing registry

• Latches the hidden feature flag once per extension-host activation.
• Creates one registry shared by the detector, inline manager, and central environment-manager router.
• Tracks saved metadata, normalized metadata identity, metadata revision, and validated-association state independently per script.
• Emits metadata and routeability transitions.
• Requires both current metadata and a validated association before shouldRoute(uri) becomes true.

Turns the existing lazy detector into a routing input only when enabled

• Reads saved PEP 723 metadata on open and save.
• Replays documents that were already open when activation occurred.
• Tracks loose local .py files for routing while retaining the existing workspace-only telemetry population.
• Withholds routing for restored dirty documents.
• Invalidates routing immediately when an edit can touch the metadata block, while body-only edits retain routing.
• Uses source-compatible offsets so BOM- and CRLF-prefixed metadata edits are classified correctly.
• Uses per-URI read generations so an older open read cannot overwrite newer saved metadata.
• Clears metadata and associations for deleted or renamed paths.

When no routing registry exists, the detector retains its existing telemetry-only listeners, coalescing, workspace filter, and event behavior.

Defines a stable metadata identity

• Uses normalized dependencies plus trimmed requires-python.
• Ignores dependency ordering, equivalent package-name spelling, and the unrelated [tool] table.
• Does not include script paths, so equivalent scripts can share cache entries.

Adds durable cache provenance

• Sidecars store bounded SHA-256 hashes of metadata identities that were proven for the cache entry.
• Same-key coalesced creators and cache-reuse callers merge their identities under the existing cache-entry lock.
• The list is deduplicated and capped at 128 entries.
• No package, requirement, script path, URI, or metadata content is stored in provenance.
• Older sidecars without provenance remain usable through conservative cache-key and Python-constraint proof; additional-package environments remain non-routeable until explicitly proven.

Makes sidecar replacement recoverable

• Serializes same-sidecar writes in-process; production callers remain protected by the cross-process cache-entry lock.
• Uses native rename with a unique backup on Windows replacement contention.
• Restores the previous sidecar after failed replacement and retains the only known-good backup if restoration is uncertain.
• Lock-held cache inspection can recover a strict, regular, size-bounded, schema-valid backup compatible with the selected base interpreter.
• Invalid, temporary, symlinked, oversized, unsupported, or incompatible-only artifacts retain normal stale/uncertain behavior.

Upgrades persisted associations from path-only values

Each current record contains:

schemaVersion
environmentPath
metadataBinding: legacy | pending(sourceIdentity) | matched(sourceIdentity)

• legacy: old string association; remains retrievable but is not automatically routeable.
• pending: the environment is proven, but saved metadata has not yet been durably matched (for example, selection while the document is dirty).
• matched: saved metadata identity and cache provenance agree.
• Future schema records are preserved rather than destructively rewritten.
• Malformed requested entries can be repaired without discarding unrelated valid/future records.

Validates associations before routing

Validation requires:

1. current saved metadata;
2. an absolute persisted executable path;
3. no active cache build/lock;
4. a resolvable environment;
5. physical ownership by the expected extension cache entry;
6. compatible Python/requires-python state;
7. a matching persisted metadata binding; and
8. sidecar proof for the current metadata identity.

Definitively stale associations are conditionally removed. Locked, transient, uncertain, or future-schema states are preserved but remain non-routeable.

Protects asynchronous validation from stale results

• Metadata revisions remain monotonic across cleared-state tombstones, preventing an old validator from winning after clear/restore of the same identity.
• A changed saved metadata identity invalidates routeability synchronously before provenance validation begins.
• Association revisions ensure unset/replacement wins over old rehydration or validation.
• Same-script rehydration and metadata refreshes are coalesced.
• Pending bindings are promoted to matched only if both metadata and association revisions are still current.
• Failed persistence never publishes successful routeability.

Adds central manager routing with explicit precedence

Priority	Manager source
1	Exact per-script project setting
2	Explicit in-session non-inline override
3	Validated inline-script association
4	Existing project/default manager setting
5	Existing cached project/global manager

• A user explicitly selecting a non-inline manager is not immediately overridden by automatic routing.
• Clearing that override re-resolves and publishes through the newly effective manager, with reserved operation ordering so a newer selection still wins.
• Inline selections are keyed by normalized script path, not containing project, so multiple scripts in one workspace can remain independent.
• Inline selections are not published as active until routeability is validated.
• Routeability changes refresh the appropriate manager and emit the existing active-environment transition.
• Invalidated metadata falls back to the existing project/default environment and removes only the inline active-selection entry.
• Existing revision checks continue preventing slow refreshes from overwriting newer selections.

User flows

Previously configured script after restart

open saved script
→ detector publishes metadata identity
→ manager loads persisted association
→ validate executable + ownership + sidecar provenance
→ registry marks association routeable
→ central environment lookup switches this file to the inline manager

Metadata edit

edit inside metadata block
→ routeability clears immediately
→ normal project/default manager remains active
→ save reads new metadata and revalidates
→ inline routing returns only if the existing association is still proven

New unassociated script

Opening a new PEP 723 script does not silently select a matching cache entry. It remains on normal routing until the future explicit setup action (PR 11/12) creates or reuses an environment and persists the association. After that, this PR provides automatic routing.

Routing and failure semantics

State	Behavior
Saved metadata + matched/proven association	Route inline
Metadata but no association	Existing project/default routing
Association but metadata unknown/dirty	Existing project/default routing
Body-only edit	Preserve current routing
Metadata edit or invalid saved block	Clear inline routing and fall back
Explicit non-inline selection	Explicit override wins
Legacy string association	Preserve/retrieve, but do not auto-route after restart
Pending binding with matching saved proof	Atomically promote to matched
Locked/transient cache state	Preserve association; remain non-routeable; retry later
Same cache path rebuilt for different provenance	Reject old routing proof
Rename/delete	Clear old path's association and routeability

Review guide

The production changes are easiest to review in this order:

1. Routing state and detection
   • routingRegistry.ts
   • activation.ts
   • metadata.ts
   • lazyDetector.ts
2. Cache provenance and durability
   • cacheLayout.ts
   • creation/reuse sections of inlineScript/envManager.ts
3. Persisted binding and validation state machine
   • persistence/rehydration/metadata-refresh sections of inlineScript/envManager.ts
4. Behavior cutover
   • envManagers.ts
   • extension.ts
   • inlineScript/main.ts

More than half of the diff is deterministic unit coverage for dirty/save/restart and async race behavior.

Tests

Coverage includes:

• activation flag latching and default-off registration;
• telemetry-only detector equivalence when routing is absent;
• open/save ordering, restored dirty editors, loose files, CRLF edits, body-only edits, rename, and delete;
• metadata-only, association-only, exact-setting, explicit-override, route-on, and fallback behavior;
• independent same-workspace script selections and batch operations;
• legacy, pending, matched, malformed, and future persisted records;
• restart rehydration and delayed routeability;
• same-path cache rebuilds and additional-package provenance;
• same-key coalesced creation and one-event telemetry semantics;
• stale metadata/association races against save, unset, and replacement;
• sidecar write contention, restoration, backup recovery, and invalid artifact handling;
• cache-clear partial failure with versioned associations; and
• public API last-known fallback/event ordering.

Validation on the final rebased tree:

• npm run compile-tests
• npm run compile
• npm run lint
• npm run unittest: 1,817 passing, 6 pending

Performance

• The feature remains hidden and default-off.
• With the flag off, no routing registry, manager, discovery timer, routing file listeners, Memento read, sidecar/cache work, or routing telemetry is added.
• The existing telemetry detector keeps its prior listener/coalescing behavior; routing checks are optional branches.
• With the flag on, metadata reads remain bounded to the first 8 KiB and occur only for opened/saved local Python files.
• Same-script validation and same-key creation are coalesced.
• Warm association validation is throttled.
• Discovery retries remain bounded and cache maintenance remains serialized only within the inline manager.

Privacy and safety

• Telemetry remains count/boolean/category based and sends no URI, path, dependency, requirement, Python version, cache key, or error text.
• Provenance uses local SHA-256 metadata identities rather than raw metadata.
• Physical cache ownership and executable checks remain fail-closed.
• Symlinked/unowned cache entries are never made routeable.
• Destructive cache behavior remains exclusively in the merged confirmation-gated cleanup lifecycle.

User impact

No default-path user impact. python-envs.inlineScripts.enabled remains undeclared and defaults to false:

• no visible setting or autocomplete;
• no new command, menu, CodeLens, picker, or status-bar surface;
• no inline manager registration;
• no routing registry, discovery, persistence, or cache work; and
• existing project/default routing remains unchanged.

For developers manually enabling the hidden flag, existing proven script associations route automatically and fall back conservatively when proof is absent. New scripts still require the future explicit setup UX.

Scope and follow-up

This PR intentionally does not implement:

• automatic setup for a newly encountered script;
• exact generated script project registration (PR 10);
• CodeLens or bulk setup UX (PRs 11-12);
• Pylance per-file interpreter support (PRs 17-18); or
• the Python debugger per-file resolver fix (PR 19).

Those later PRs can build on this guarded routing layer without changing its validation contract.

[Heejae Chang (heejaechang)]

🔒 Automated review in progress — Heejae Chang (@heejaechang) is auto-reviewing this PR.

[Heejae Chang (heejaechang)]

Verification: The relevant tests could not be fully run in the isolated environment; this review is not fully verified.

[Heejae Chang (heejaechang)]

Verification: The relevant tests could not be fully run in the isolated environment; this review is not fully verified.

[Heejae Chang (heejaechang)]

Approved via Review Center.

[Eleanor Boyd (eleanorjboyd)]

Thanks for the thorough work here. I found three correctness issues that need addressing before approval:

1. InlineScriptRoutingRegistry.setMetadata() preserves validatedAssociation when the metadata routing identity changes. This leaves the old association routeable while async provenance validation for the newly saved metadata is pending, so callers can temporarily receive the previous inline environment. Please invalidate association validation synchronously whenever the metadata identity changes.

2. When a non-inline script override is cleared, setEnvironment() removes the override but continues resolving and publishing through the old manager/key. It also deletes the now-relevant inline last-known selection, so timeout-based API callers can receive undefined and no later manager event is guaranteed to correct it. Please re-resolve and publish through the manager that becomes effective after override removal.

3. The cache creation/reuse path rejects entries when the selected base interpreter version has changed or the environment no longer satisfies requires-python, but cached and rehydrated persisted-association validation does not apply those checks before restoring routeability. A same-path interpreter upgrade can therefore keep an association routeable that fresh cache validation would reject. Please apply the same base-version and compatibility invariants to persisted associations.

Test note: triggerSavedMetadataChange() both publishes through the registry and directly invokes the private handler. It would be valuable to exercise only the public event path so broken production wiring cannot be masked.

Written by an AI agent on behalf of Eleanor.

[Heejae Chang (heejaechang)]

Verification: The relevant tests could not be fully run in the isolated environment; this review is not fully verified.

[Stella Huang (StellaHuang95)]

Addressed the three correctness issues from review 4985022926 in aed8aee:

1. InlineScriptRoutingRegistry.setMetadata() now preserves association validation only when the normalized routing identity is unchanged. A changed identity synchronously emits the routeability-off transition before asynchronous provenance validation starts; equivalent metadata spelling/order remains routeable.

2. Clearing an explicit non-inline override now reserves the inline selection operation before awaiting the old manager, removes the override, re-resolves the effective manager, and publishes its environment instead of undefined through the old manager. The reserved operation is abandoned if any newer inline selection begins, so the handoff cannot overwrite newer user intent. Tests cover both the normal override-clear transition and the concurrent newer-selection race.

3. Warm and restart association validation now require a valid sidecar's recorded base release to match the resolved runtime and the current requires-python constraint to accept it before direct retrieval or routeability. When strict proof fails during selection, the known current identity is retained as non-routeable pending rather than degraded to broad legacy behavior. Existing future/unavailable sidecar preservation remains unchanged.

I also added a focused test that drives validation solely through routingRegistry.setMetadata() and the manager's constructor subscription. The existing private-handler tests remain for deterministic race control, but production event wiring is now independently covered.

Validation: compile-tests, bundle, lint, and full unit suite — 1,806 passing, 6 pending.

[Heejae Chang (heejaechang)]

Verification: The relevant tests could not be fully run in the isolated environment; this review is not fully verified.

[Heejae Chang (heejaechang)]

GitHub cannot anchor PR review comments to unchanged lines in the diff. Falling back to a general PR comment for src/managers/builtin/inlineScript/envManager.ts:L227.

Warning · Non-blocking recommendation

routingRegistry defaults to a new isolated registry, although routing requires the activation-scoped registry shared with the detector and PythonEnvironmentManagers. Make this dependency required (and provide it explicitly in tests) so a composition mistake cannot silently disconnect association validation from central routing.

[Heejae Chang (heejaechang)]

Verification: The relevant tests could not be fully run in the isolated environment; this review is not fully verified.

[Heejae Chang (heejaechang)]

Verification: The relevant tests could not be fully run in the isolated environment; this review is not fully verified.