feat(pull): /chat on the durable queue for pull pilots (#3127)

[obasilakis]

Stacked on #3124 (base: feature/3114-pull-route-interactive). Review the top 3 commits; after #3124 merges this PR is retargeted to dev.

Summary

The last push path on a pull-pilot agent. POST /api/agents/{name}/chat is called by MCP chat_with_agent, the agent-to-agent connector tool, and the trinity chat CLI. On a pilot it is now admitted onto the durable queue and claimed by a worker, so every interactive trigger is pull-owned on a pilot. The web UI chat panel already went through POST /task, which #3124 routes.

• Memory on a pilot is one Claude conversation per (agent, user) (operator decision on feat(pull): UI /chat on the durable queue for pull pilots #3127). Turns run through session_turn_service.run_resumable_turn with key session:chat:<chat_sessions.id>, resuming chat_sessions.cached_claude_session_id. That gives the ResumeLock, persist_session, and a cold retry on resume-not-found. Off a pilot /chat keeps the agent container's single shared session.
• Admission order is unchanged: depth guard, idempotency begin, breaker read. A pilot then skips the slot acquire.
• Post-turn work that /chat did inline runs in the caller after the terminal: assistant chat_messages row (secrets scrubbed), activity closes, the same response shape, idempotency complete, a timeout receipt + 504, and ResumeLockBusy → FAILED + 429.
• GET /chat/history on a pilot serves the caller's session from the database. DELETE /chat/history closes the sessions /chat used and forgets their ids.
• New column chat_sessions.cached_claude_session_id (SQLite migration + Alembic 0086), which joins the session reaper's keep set. execute_task(chain_depth=) keeps the depth on a cold-retry row.
• Client disconnect: tested against the app's middleware stack, the /chat handler is not cancelled when the client goes away. The MCP 25s abort therefore leaves the queued turn running and its receipt valid.

Live verification (isolated sibling stack, real Claude turns, 3-worker pilot + non-pilot)

• /chat on the pilot: 3 turns from one user were pulled on one Claude session and recalled earlier numbers ("17, 29"). A second user got "NONE" on their own key and session. History returned each caller's own messages. A reset made the next turn start cold, and history is empty after it.
• Non-pilot: /chat was pushed with memory kept, as before.
• The same run covered feat(pull): route interactive producers onto the durable queue on pull pilots (#3114) #3124's Workspace paths: portal sync/async + memory, the stream hold, Stop while queued/running/right after claim, room serialization, and an unclaimed turn → FAILED "Agent at capacity". The issues it found are fixed on feat(pull): route interactive producers onto the durable queue on pull pilots (#3114) #3124 (a04d6e8ce) and here (883656752).

Known gaps

• On a pilot a /chat turn gets the task-mode platform prompt, where push gives it the chat mode.
• output_tokens is not recorded on pulled /chat messages.
• GET /chat/session still reads the agent's shared session.
• After a cold retry, the MCP receipt lookup may point at the failed first row.

Test Plan

• cd tests && pytest unit/test_3127_pull_chat.py -v (21 tests; 11 key-behaviour mutations + the reset mutation go red)
• Flipped pins: test_1766, test_2391, test_3114_pull_route_interactive, test_2610
• Combined feat(pull): route interactive producers onto the durable queue on pull pilots (#3114) #3124 + feat(pull): UI /chat on the durable queue for pull pilots #3127 suites: 481 passed
• check_alembic_heads: one head (0086)
• /cso --diff: 0 findings

Fixes #3127

🤖 Generated with Claude Code

[github-actions]

✅ Alembic head check clear — merging this PR into dev leaves one head (0089_supersede_queue_flood_backlog).

Previously flagged; resolved.

Advisory — this check does not block merge. · head_sha: 34830370fe66b46fabb2592a7d85842516df1e86 · run

[github-actions]

✅ Nightly unit-suite clean when this PR is merged into dev, all 3 seeds (head_sha: 34830370fe66b46fabb2592a7d85842516df1e86).

[github-actions]

⚠️ Live-instance suite skipped — merge conflict against dev.

Resolve by merging dev locally and pushing the result; the next nightly re-tests.

[vybe]

merge-train (2026-10-04): not on this train. It rides the next one once fixed. This PR was not validated today because the diff will change with the rework below.

The branch is 45 commits behind dev and no longer merges:

• Conflicts in src/ against dev: db/migrations.py, routers/chat.py, routers/paid.py, services/capacity_manager.py, services/pull_pilot.py, services/task_execution_service.py, shared_sessions/service.py, and the portal files PortalConversation.vue, portalWork.js and portalWork.spec.js. These are a re-apply over code that moved, so they are yours to resolve rather than a train fix.
• add/add conflicts in tests/unit/test_3114_pull_route_interactive.py and test_3114_pull_route_workspace.py: feat(pull): route interactive producers onto the durable queue on pull pilots #3114 has since been squash-merged, so the stacked copies collide with dev's.
• Alembic fork (the red alembic-head-watch): 0086_chat_session_claude_id has down_revision = "0085_ent720_email_identity", the same parent as dev's 0086_metric_points_restatement. dev's head is now 0088_skill_gate_requests, so after merging dev this revision becomes 0089_chat_session_claude_id on top of it. python3 scripts/ci/check_alembic_heads.py src/backend/migrations/versions should report one head.
• feat(skills): gated skills — the approval comes before the agent sees the request (abilityai/trinity-enterprise#751) #3208 (gated skills) merged today and touches 17 of this PR's files, including chat_execution_service.py, dispatch_admission_service.py, public_chat_service.py, task_execution_service.py, message_router.py and routers/internal.py. Its gate reads request_text at the /chat and /task admission seams, so the new durable-queue /chat path needs to pass through them.

I've set status-needs-fix; your next push clears it.

[vybe]

merge-train: not on today's train. It rides the next one once fixed.

Merged with current dev, regression diff fails tests/unit/test_ent751_gate_guards.py::test_every_producer_passes_the_requesters_own_words_or_is_named_unwrapped. The guard landed on dev after this branch was cut, so the PR head alone passes.

producers that compose a message without `request_text=` and are not named unwrapped:
[('services/chat_execution_service.py', 'run_pulled_chat_turn')]

The new run_pulled_chat_turn calls session_turn_service.run_resumable_turn(...) (~chat_execution_service.py:1007) with neither request_text= nor gate_checked=True. Picking one is a skill-gate (ent#751) decision, which is why this went back to you instead of getting a mechanical fix:

• If the /chat router's skill_gate_service.enforce(...) (~:2374) always runs before the pull branch, pass gate_checked=True, as the push path does.
• Otherwise pass request_text= with the same joined message, user_message and system_prompt that the router gates on.

Adding it to _UNWRAPPED would be wrong. /chat carries the requester's own words, which is exactly what the gate must read.

Reproduce: merge origin/dev into the branch, then run pytest tests/unit/test_ent751_gate_guards.py.

[AndriiPasternak31]

Thanks for this. The pull routing is clean: I couldn't find a double-execution path (run_chat_turn branches only on capacity is None, and execute_task re-decides with the same env-only predicate). The only caller-side terminal write is on a never-enqueued RUNNING row, so it can't race the token-gated sink, and the slot and idempotency accounting balance. Tier 1 is green and check_alembic_heads passes. I ran test_3127 + the flipped pins (164 passed) and the 1804/1578/2806/2842/2889/ent751/3114 guards (297 passed).

Two things I'd like fixed before merge. Both are invisible to the suite because dispatch_and_await_terminal is mocked in every turn test:

1. Skill gate runs twice on a pilot /chat (ent#751). The comment at chat_execution_service.py:1011 says "/chat runs no admission-seam gate", but admit_chat_request calls skill_gate_service.enforce at dispatch_admission_service.py:326, before the pilot branch at :387. Without gate_checked=True, execute_task's backstop gates again with a requester rebuilt from the row (requester_for_dispatch), and that requester never has is_person=True. I checked this with the real enforce: the admission requester self-approves the owner, and the backstop requester does not. So on a pilot, an approver's own gated request turns into a pending approval: the row is SKIPPED, a second ask is raised, the caller gets a 202, and the user message has no reply. The pilot branch also returns before audit_self_approved (:431). Fix: pass gate_checked=True (keep request_text), call audit_self_approved in the pilot branch, and assert gate_checked in test_pilot_turn_dispatches_through_the_queue. This is the first branch of vybe's 10-05 note.

2. Agent-to-agent /chat (trigger agent) gets no claim priority and no claim budget. "agent" is not in INTERACTIVE_TRIGGERS, so _CLAIM_WAITING_TRIGGERS (task_execution_service.py:720/811) skips the phase-1 wait. The turn queues behind batch work, and the only bound is timeout + 120s from enqueue. With a deep queue that ends in the 504 queued_timeout receipt while the row is still queued. It then runs for nobody: no assistant chat_messages row and no cached Claude id. The collaboration activity also stays started until the 120-min backstop, because build_pull_queue_payload sends collaboration_activity_id=None. That's the #1804 symptom. The a2a note in pull_pilot.py states the rule: a blocked caller earns priority and the budget. Could the pulled /chat opt into claim semantics explicitly (rather than by trigger), and could the collaboration activity id ride the payload so the sink closes it? A test with the real adapter for trigger agent would pin it.

Smaller items, non-blocking:

• Post-turn work runs in the caller, so it's lost on a backend restart or a 504. Issue scope item 3 said "moves to the terminal", and #3227 puts post-turn delivery in the sink. No textual conflict between the two PRs, but worth converging once #3227 lands (carry chat_session_id and collaboration_activity_id in the metadata).
• Same-user concurrency: the ResumeLock waits 30s and then the call gets a 429 and the row is FAILED. Agent keys resolve to the owner, so every agent of one owner shares one session with the target, and parallel chat_with_agent calls fail where push serialized them. Consider a pilot lock wait of at least one turn, as rooms do.
• cached_uuid is read before the lock (:1005), so two concurrent first turns both run cold and the second overwrites the first's id.
• Lock-busy path: the update_execution_status bool is ignored, no agent.task.failed is emitted (MCP's receipt goes out at 25s and this FAILED lands at 30s), and the 429 has no X-Trinity-Error-Code: capacity.
• Backlog full on a pilot /chat maps to 503 with no code, because "Agent backlog full…" doesn't match "at capacity". Push answered 429/capacity.
• The Chat tab writes into the same chat_sessions row, so "New chat" in the UI resets MCP/CLI memory, and the owner reset closes Chat-tab sessions.
• Docs: the chat_with_agent description in mcp-server/src/tools/chat.ts:395-399 still says the session is shared by every caller and restarts on a model change. persistent-chat-tracking.md, mcp-orchestration.md and agent-to-agent-collaboration.md aren't updated. The PR body and the CSO report still say Alembic 0086 and "stacked on #3124".
• Alembic: #3255 and #3256 also add 0090_* off 0089_supersede_queue_flood_backlog. The tables are disjoint, so this is a mechanical re-parent for whoever lands second.