test: promote three QA regression anchors (QA-716 / QA-627 / QA-681)

[kriszyp]

What / why

Three QA regression anchors promoted out of the qa-explorer bank as committed integration suites. Test-only: the diff is ten new files — three suites, their three fixture components, and one fixture resource — and zero modifications to anything tracked.

One line per spec, with the originating QA id:

• integrationTests/resources/qa716-lingering-write-commit.test.ts (QA-716 / P-493) — pins PR fix(txn): don't drop writes staged while read iterators defer the commit #1860 ("don't drop writes staged while read iterators defer the commit"): writes staged across three tables in one request transaction, behind a paged secondary-index iterator the handler returns without draining or closing, must be durable and index-consistent the moment the request is acked — not eventually.
• integrationTests/security/qa627-cli-user-auth-separation.test.ts (QA-627 / P-409) — pins CLI: add_user/alter_user unusable in CI/CD — auth creds and new-user payload share username=/password= #1872, fixed by PR fix(cli): separate transport auth from operation payload for add_user/alter_user #1873: the CLI resolves HTTP Basic credentials separately from the add_user/alter_user payload, and per source as an atomic pair.
• integrationTests/mqtt/qa681-mqtt-shared-subscriptions.test.ts (QA-681 / P-460) — pins that MQTT $share subscriptions are explicitly refused with a SUBACK reason code rather than accepted as an inert filter, that ordinary fan-out beside them stays complete and duplicate-free, and that a durable session loses nothing across an abrupt mid-stream drop.

Why QA-716 is additive to unitTests/resources/lingeringWriteCommit.test.js

That unit test shipped with the #1860 fix and covers the basic single-table case against the transaction object in isolation. This anchor drives the same mechanism through the whole request path and combines it with the factors a real fulfillment endpoint hits at once: writes staged across three tables in one request transaction, a paged secondary-index iterator the handler never drains or closes, TTL eviction racing the deferred commit, and 4-way worker concurrency. The fixture's FulfillPage is where the shape lives — it pulls a page and returns with the cursor simply dropped.

Both are RocksDB-only in substance, since LMDB never defers a commit on open read transactions, so the suite is parameterized on HARPER_STORAGE_ENGINE and the LMDB run is a no-delta control. Q3 asserts that the long-transaction monitor actually reached the abandoned iterator, via #1860's release-only warn line — on RocksDB only, because LMDB correctly never logs it. Without that assertion the arm was blind: dropping the lingering transaction from write supervision, or renaming storage.maxTransactionOpenTime, would have left it sleeping and then re-reading writes Q1 had already proved durable.

QA-627's polarity was inverted, deliberately

The banked candidate asserted the pre-fix defect — that the CLI must 401 because it authenticated as the user being created. Re-run as banked on current main it fails 2 of 5 arms, both with expected the CLI to fail (auth collision) but it exited 0: red there means the fix works. Promoting it as a defect repro would commit a test that only goes green if #1873 is reverted, so the arms assert the shipped contract instead — env-var credentials, auth_* args and target-URL userinfo each winning the auth leg over the payload, the atomic-pair rule, and the legacy username=/password= fallback surviving for operations where those args genuinely are the credentials.

For the human reviewer

• One PR across three directories. The promotion rules ask for one subsystem per PR; QA-716 and QA-627 are both shipped-fix anchors, which is the accepted cross-directory theme, and QA-681 was bundled rather than opening a second PR because the three files share no code and are read independently. Happy to split QA-681 out if you would rather review it separately.
• QA-681 has no ratifying issue, and that is the judgment call worth your attention. It is a clean negative pinning behaviour that is real but undocumented: Harper has no $share parsing at all, so the literal prefix fails resource resolution and the SUBACK refuses. The promotion gate routes it to the human lane for exactly that reason. The question is whether refusing $share is the contract Harper wants to be held to, or merely today's behaviour — a test that pins it makes it the former.
• describe_all with plain username=/password= is a compatibility anchor, not a fix anchor, and the file says so: that arm passed before fix(cli): separate transport auth from operation payload for add_user/alter_user #1873 too. It fails only if the legacy fallback is dropped outright.
• Two claim scopes were narrowed to what the instruments actually prove, and the code says so at each site. search hides a row the moment expiresAt passes, independently of the sweep (resources/Table.ts), so QA-716's expiry assertion says an expired row is no longer returned rather than that a sweep ran.
• QA-716's TTL arm deliberately makes no index-level claim, and says so where the assertion used to be. search_by_value materializes its hits and drops any whose base record is gone, so once the base dump is empty an index query is empty too, whatever the index actually holds — a check there could not fail. Pinning sweep-time index cleanup needs a raw-index read the operations API does not expose. Worth knowing: the same limit applies to the phantom oracle in the committed integrationTests/database/eviction-secondary-index.test.ts:310-315, whose residual index entries: 0 and phantom index entries: 0 assertions are incapable of failing for the pure-orphan case they name. That is pre-existing and filed separately, not introduced here.
• Three fixed grace windows survived on purpose, in QA-681 Q3/Q4 and QA-716 Q3, and each guards a claim about an absence — no message ever reaches a refused subscriber; no write is discarded once the long-transaction monitor has run. There is no positive event to converge on for those, so elapsed time is the instrument. Every positive claim converges instead.
• Declined from the pre-push review, with reasons:
  • "authorizeLocal is never disabled, so the CONTROL arm will fail" (raised as a blocker). Refuted twice over: the harness passes --AUTHENTICATION_AUTHORIZELOCAL=true as a CLI arg, never as process.env, so security/auth.ts's env branch is undefined and the HARPER_SET_CONFIG value from config: decides — and empirically the CONTROL arm returns 401 on every run. That arm exists precisely to catch this if it ever changes.
  • Add a close listener to the MQTT connect handshake. The 10s cap already converts a silent handshake drop into a clear failure, and a close listener risks a false rejection in mqtt.js's normal event ordering — a worse trade for a test that must not flake.
  • Extract the readiness poll into a shared awaitFixtureReady() helper. Every committed suite in integrationTests/ inlines its own poll; a promotion PR is the wrong place to introduce a new cross-suite API, and three similar blocks beat a premature abstraction.
  • Delete the file headers as "PR history". They are this suite's own convention for promoted anchors — qa649-mqtt-restart-wedge.test.ts, txn-overtime-atomicity.test.ts, overtime-multi-write-atomicity.test.ts and eviction-secondary-index.test.ts all carry one — and each records which fix the anchor pins and what the pre-fix failure looked like, which is the "why" the code cannot express. The inline narration flagged alongside them is gone.
• Not covered here: the field-stripping half of fix(cli): separate transport auth from operation payload for add_user/alter_user #1873 (auth args never serialized into the request body). It has no server-observable effect — add_user ignores an extra auth_username field — so asserting it from an integration test would be theatre. unitTests/bin/cliOperations.test.js already owns it.

Verification

Everything below ran in the worktree on current main (ee46a6acf) after npm run build (exit 0), because integration tests run against dist/.

suite	result
qa716-lingering-write-commit.test.ts	4 pass / 0 fail
qa716-lingering-write-commit.test.ts (HARPER_STORAGE_ENGINE=lmdb)	4 pass / 0 fail
qa627-cli-user-auth-separation.test.ts	8 pass / 0 fail
qa681-mqtt-shared-subscriptions.test.ts	4 pass / 0 fail
all three together	16 pass / 0 fail

• npx prettier --check and npx oxlint --format stylish --deny-warnings are clean on all three specs and the fixture resource. oxlint was confirmed armed rather than merely silent, using a non-underscore probe (qaProbeUnusedImport) that fired as expected and was then removed — oxlint's no-unused-vars exempts underscore-prefixed identifiers, so an _-named probe proves nothing.
• CI note: the Unit Test (Node.js v22) leg failed on unitTests/resources/longLivedTransactions.test.js — "names a chain link reachable only through the root under its own native id", asserting /state: [^,]*active/ and getting state: source-apply. That is pre-existing and unrelated: this diff adds files only under integrationTests/, every unit script globs unitTests/**, Unit Test (Node.js v24) passed on the same commit, and the identical failure hit main's own CI run for Report the long-lived transaction holder behind a wedged commit #2473, the commit that introduced the test. The cause is the 10-line-per-pass cap in reportLongLivedHolder deferring a holder past the single tick during which active is still in the state list, so a loaded suite intermittently reports the next state instead.
• The qa-explorer promotion gate reports 0 hard rejects across all three: no vacuous-assert, abs-path, restart-fixture, bare-sleep, raw-restart, format or lint.

The three fixtures

Each suite pre-installs its own component, so none of them restarts workers.

• QA-716 needs a custom resource, so its component declares jsResource alongside the schema, and its schema carries the three tables the staged writes span: Orders.status as the indexed attribute under test, Inventory as a second table in the same transaction, and Reservation as a third with an 8-second TTL — 8 rather than 3 because search stops returning a row the moment expiresAt passes, so the window is wall-clock time from the write.
• QA-681 is the only one that enables mqtt; its Events table is a topic namespace, and non-retained publishes to Events/* give a transient stream with no per-key coalescing, which is what makes delivery countable.
• QA-627 exercises the operations API through the CLI rather than any table, so its component and FirstTable exist only to give the readiness probe a GET-able route.

Repairs made before promoting

The banked snapshots carried five defects that would have shipped as coverage they did not have. They were fixed rather than carried in:

• QA-716's TTL arm returned green after logging a skip when no Reservation rows were present — the exact shape where a lost upstream write makes the eviction assertions trivially true over an empty table. It now asserts a floor, and seeds its own bucket so its expiry window is not shared with another arm's network round trips.
• QA-681's four arms each returned green after logging SKIPPED (harness) if the MQTT probe failed, so a run that never connected reported coverage it had not taken. They now fail. The same applied to all three suites' readiness polls, which proceeded silently after their 120s deadline.
• QA-681's durable-session catch-up was logged and never asserted, and could not distinguish "the session recovered its missed messages" from "the drop never cut delivery". It now asserts the drop cut delivery first, then that the union of before-drop and recovered messages covers everything published — the claim that is stable across shard speeds, unlike the exact split.
• QA-681's recovery collector was attached after await connect(). A resuming durable session replays its QoS-1 backlog immediately after CONNACK and mqtt.js acknowledges those PUBLISHes before the awaited continuation can install a listener, so the new recovery assertion would have been flaky against a perfectly healthy broker. The collector is now attached synchronously at client creation.
• QA-627's arms were decided by the developer's own machine. Every credential source the CLI consults outranks the legacy payload fallback, so a saved harper login token in ~/.harperdb/credentials.json, a repo-root .env (cliOperations() calls dotenv.config() before resolving auth), an inherited HARPER_CLI_USERNAME, or a CI runner's OIDC identity would each silently decide what the fallback arms resolved to. The child now runs with a throwaway HOME and cwd and an env scrubbed of all four.
• QA-627's lone-username= check could not detect a revert: pre-fix the CLI authenticated as a nonexistent user (401) and post-fix it sends no credentials (also 401), so both polarities matched. It now runs with a complete env pair configured, where the two differ observably.
• Three arms shared state that made them race each other. QA-716's TTL arm read rows another arm's network round trips could expire; QA-681's Q3 shared a topic with Q1, whose last row is that topic's current record and is yielded to a new subscriber by a detached iterator that can lose the race to SUBACK; and Q4's drop boundary raced the next publish because end(true) does not wait for the socket to be down. Each arm now owns its state — Q3 subscribes through a $share filter wrapping its own topic — and converges on the transition it depends on.
• QA-716's post-eviction index assertion could not fail once the base dump was empty — both go through the same materialization. It is gone, with a comment explaining why, rather than left reporting coverage it did not have.
• QA-627's target= URL arm depended on the shape of the ambient admin password. The credential reaches new URL() inside bin/cliCredentials.ts, which percent-encodes userinfo that extractTargetCredentials then returns undecoded, so @, :, ; or = in HDB_ADMIN_PASSWORD would have reported a fix(cli): separate transport auth from operation payload for add_user/alter_user #1873 regression that never happened. The arm provisions its own super_user with an alphanumeric password instead. That decoding gap is a real pre-existing product bug — a target=https://user:p%40ss@host credential cannot work — and is filed separately rather than worked around silently here.
• QA-681's resumed-session check fought QoS-1's at-least-once contract twice. It first forbade duplicates while allowing extra deliveries — two assertions two lines apart that contradicted each other — and then, once duplicates were permitted, still waited on a delivery count: a resume replaying an unacked seq 14 alongside 15–28 reaches the expected total while seq 29 is still in flight, so the poll exits early and the run fails a session that recovered everything. The wait is now on sequence coverage, which is duplicate-tolerant by construction. The survivors keep their exact-count assertion — that one is the anti-redistribution claim, and it is safe because they never reconnect.

One latent false-green in the banked QA-627 is gone with the inversion: its NOT-A-WORKAROUND arm asserted only a non-zero exit, and on current main it "passed" because the CLI failed with User qa627_svc_app already exists — a 409 left by the previous arm, not an auth collision.

Refs #1860
Refs #1872

Reviewed and authored by Claude Opus 5.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JZPy6UYZtwxhzLgimHWbGD

_{Review-Coverage: authored=claude; ran=codex,gemini; declined=cursor-grok,cursor-composer,domain; rounds=8 @ ca6d3ff}

_{Human-Review-Need: 3 @ ca6d3ff}

[gemini-code-assist]

Code Review

This pull request introduces several integration tests as regression anchors, covering MQTT shared-subscription semantics, lingering write commits with abandoned read iterators, and CLI user authentication separation. The reviewer's feedback focuses on improving the robustness of the MQTT integration tests by ensuring proper cleanup of connections in finally blocks to prevent leaks during failures, and resolving a potential Temporal Dead Zone ReferenceError in the connection helper function.

[claude]

Reviewed; no blockers found.