Skip to content

Synchronize browser specs with capybara-lockstep - #326

Merged
HamptonMakes merged 7 commits into
mainfrom
hampton-postmerge-browser-stability
Nov 13, 2026
Merged

HamptonMakes merged 7 commits into
mainfrom
hampton-postmerge-browser-stability

Conversation

@HamptonMakes

@HamptonMakes HamptonMakes commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

What changed and why

Browser specs can advance while requests or Turbo updates still change the page. Hampton asked for shared browser synchronization instead of a JavaScript snapshot inside the math test.

Add the test-only capybara-lockstep gem. Load its snippet before application scripts in the host and engine layouts, and cover ActiveAdmin's separate layout with a test-only head wrapper. Use its Rack middleware for server requests. A shared test hook tracks received Turbo Streams through their final render promise, including cancellation and errors.

The math assertion now reads ordered text in Ruby inside Capybara's retry loop. Checkbox assertions also re-query named task fields after saved versions replace the body. Exact count/text, stored source range, formula exclusion, and reload checks remain. Tests that deliberately inspect held requests or pointer gestures explicitly use browser_sync: :manual; their assertions remain active and automatic mode is restored afterward.

Keep the gem's default timeout logging and next-command synchronization. A document navigation can discard the old page's probe; a log entry does not establish readiness. Feature assertions still require the expected result. The stream cleanup regressions explicitly await the idle callback and fail if work leaks. Production navigation and visual styles are unchanged.

Evidence

  • Final current-head CI passed all fifteen checks on e1a7c82a. Codex review completed on that exact commit with no findings, and GitHub reports no merge conflicts.

  • The real Action Cable regression fails with the shared Turbo hook removed and passes with it installed. It checks the delayed render and actual comment highlights.

  • Real fetch/callback, cancelled stream, and rejected renderer cases exercise the shared integration. The admin browser spec verifies that its page includes the synchronization snippet.

  • Early strict-timeout probes failed across sign-in document replacements. A time-origin probe confirmed the reload. The final implementation uses the package's normal navigation-tolerant behavior; it does not change login or theme behavior to satisfy the tests.

  • The reliability record preserves the original Preserve recovered draft titles and numbered title links #311/HOOD-6816 Keep linked comment windows open after rendering #324 failures, the natural math-race reproduction, and the limits of their diagnostic retries. The separate caret-position failure remains unexplained. The cancelled historical shards remain incomplete.

  • The first full CI run exposed two stale checkbox handles after body replacement. Both reproduced locally at seed 49204. The corrected current-field assertions retain the saved-content, version, attribution, and provenance checks.

  • The next full run passed fourteen checks and failed a synthetic folder drag. Its trace failed at dragend; the screenshot already showed the completed move. That case passed once in isolation locally. The helper now dispatches drop and dragend in one browser turn before synchronization can wait for navigation; folder, placement, and final-path assertions remain.

How to try it

From this branch, use a dedicated MySQL database:

bundle install
RAILS_ENV=test DATABASE_URL=mysql2://root@localhost/coplan_browser_sync_tryout bin/rails db:create db:schema:load
RAILS_ENV=test DATABASE_URL=mysql2://root@localhost/coplan_browser_sync_tryout bundle exec rspec spec/system/browser_synchronization_spec.rb spec/system/math_rendering_spec.rb spec/system/admin_spec.rb

Expect delayed fetches and received live updates to finish before their checks, cancelled/failed streams to release synchronization, and all math/admin cases to pass. The command starts its own test server. See the browser testing guide for manual mode, timeout behavior, and work that is not automatically tracked.

Testing

  • bundle exec rspec spec/system/folders_workspace_spec.rb:637 spec/system/browser_synchronization_spec.rb --seed 20822 — all three spring-loaded-folder cases and four synchronization cases passed after the gesture correction.

  • bundle exec rspec spec/system/checkbox_spec.rb spec/system/browser_synchronization_spec.rb --seed 49204 — 17 examples, zero failures after the checkbox correction.

  • Final merged-base check: bundle exec rspec spec/system/browser_synchronization_spec.rb spec/system/admin_spec.rb spec/system/math_rendering_spec.rb spec/system/callouts_spec.rb spec/requests/sessions_spec.rb --seed 41213 — 34 examples, zero failures, isolated MySQL database.

  • Focused integration checks during development: 167 inline-editor/reaction/live-delivery/review examples passed with diagnostic logging, plus 56 editor recovery/toolbar/mode examples. These preceded the final default-timeout configuration; current-head CI validates the complete suite.

  • Changed Ruby lint and git diff --check passed.

  • A full local suite was avoided because other workers share this Mac. Full coverage runs in CI on the published head. No UI controls or visual styles changed, so additional mode screenshots do not apply.

  • Exercised the real browser, fetch, and Action Cable paths.

  • Domain behavior and API contracts are unchanged.

  • Recorded focused checks and why the local full suite was avoided.

  • Test infrastructure only; product navigation and visual styles are unchanged.

  • Runnable tryout and explicit synchronization limits are documented.

Generated with Codex

@HamptonMakes

Copy link
Copy Markdown
Collaborator Author

🤖 @codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-12T22:54:54.032827Z e1a7c82 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: f818bd229f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@HamptonMakes

Copy link
Copy Markdown
Collaborator Author

🤖 Cleanup status for f818bd229f4aea44eb331e8ff80a023bc79c0694:

  • The math highlight assertion now reads ordered text in one browser turn inside Capybara's retry loop. It retains the exact count/text, source-range, formula-exclusion, and reload checks. Product code is unchanged. This is separate from Hampton's requested bin/ci replacement in Run local CI with parallel_tests #317.
  • The unchanged math case reproduced naturally on the fifth fresh-process run (seed 41213). After the patch, ten fresh processes (41209–41218) and the combined math/list specs (37 examples) passed. Lint and diff checks passed.
  • Codex review completed with no findings on this exact head. The branch has no merge conflicts; the PR remains draft.
  • Current-head CI: all test steps passed, including both database suites, all four browser shards, all stress cohorts, and voice cleanup. CI is not green: report-upload steps failed in both database jobs, browser shard 4, and stress cohort 0; GitHub still reports the overall run in progress. Other jobs have completed their steps but their final job conclusions remain pending. The job-log endpoint returns HTTP 502, so the underlying upload error is not yet verified. No rerun was requested and no checks were weakened.
  • The original Preserve recovered draft titles and numbered title links #311 caret-position failure remains unexplained. Five isolated runs and three runs in its exact original predecessor order passed; those results do not establish its cause or prove a product fix. The single failed-job-only retries for Preserve recovered draft titles and numbered title links #311 and HOOD-6816 Keep linked comment windows open after rendering #324 passed unchanged, while both historical workflows still conclude cancelled because their originally cancelled shards were not rerun.

The PR body includes dedicated-database commands for trying the corrected math spec. Engineering handoff remains pending the CI upload/finalization problem; this is not a merge or readiness claim.

@HamptonMakes

Copy link
Copy Markdown
Collaborator Author

🤖 CI follow-up on unchanged head f818bd229f4aea44eb331e8ff80a023bc79c0694:

Attempt 1 finalized with four failed report uploads and seven cancelled jobs. All test steps passed before that finalization. The now-available logs/annotations identify the same infrastructure error in all four failures: Failed to CreateArtifact: Failed to make request after 5 attempts: Unexpected token '<', "<!DOCTYPE "... is not valid JSON. The MySQL log also confirms 2,693 examples, zero failures, before the artifact-service requests failed. This establishes the upload failure, without claiming a confirmed global GitHub incident.

Requested one --failed retry at 21:04 UTC. Attempt 2 includes all eleven unsuccessful jobs: the four upload failures and seven cancelled jobs. The successful lint check was not rerun. These jobs are queued; CI is not green yet. No code or workflow change and no duplicate blanket retry.

The separate caret-position failure remains unexplained. Codex review remains clean on this head; the PR stays draft pending completed checks and Hampton's acceptance.

@HamptonMakes

Copy link
Copy Markdown
Collaborator Author

🤖 Final cleanup summary for f818bd229f4aea44eb331e8ff80a023bc79c0694:

  • The math browser spec reads the ordered highlight text in one browser turn inside Capybara's retry loop. Exact count/text, stored source range, formula exclusion, and reload checks remain. Product code is unchanged. The original stale-element failure reproduced naturally before the patch; ten fresh-process runs and 37 combined math/list examples passed afterward. Lint and diff checks passed.
  • Current-head CI is fully green: attempt 2 completed successfully at approximately 21:15 UTC. All fifteen PR checks passed, including both database suites, every browser shard, stress cohorts, voice cleanup, and report uploads. The single authorized unsuccessful-job retry covered all four artifact-service failures and all seven cancelled jobs. No additional retry was needed; no checks were weakened.
  • Attempt 1 remains recorded as failed. Its four CreateArtifact failures received HTML where the artifact service expected JSON after five attempts. This was an upload infrastructure failure; it does not establish a confirmed global GitHub incident.
  • Codex review completed with no findings on this exact head. There are no merge conflicts. The PR remains draft pending Hampton's acceptance; no merge is authorized. The PR body has the dedicated-database tryout commands and expected math-spec result.
  • The Preserve recovered draft titles and numbered title links #311 caret-position failure remains unexplained. Local isolated/exact-predecessor passes and the unchanged diagnostic retry do not prove its cause or a product fix. The original Preserve recovered draft titles and numbered title links #311/HOOD-6816 Keep linked comment windows open after rendering #324 workflows still conclude cancelled because their original cancelled shards were not rerun. This PR's successful full coverage does not retroactively change that historical coverage.

This follow-up remains separate from Hampton's requested open-source bin/ci replacement in #317, which is independently ready for review.

@HamptonMakes
HamptonMakes marked this pull request as ready for review November 11, 2026 21:40
@HamptonMakes HamptonMakes mentioned this pull request Nov 11, 2026
5 tasks done
…wser-stability

* origin/main:
  Use parallel_tests for local CI
  Run local CI checks and RSpec in parallel
…wser-stability

* origin/main:
  Keep table content when saving callouts
  Preserve summary callouts in list containers
  Add a [!SUMMARY] callout for TL;DRs
  Wait for math comment highlights after broadcasts
  Preserve callout syntax across editor saves
  Preserve callout boundaries and deck contrast
  Re-run CI (browser timeout in an unrelated spec)
  Render GitHub-style callouts for notes and warnings

# Conflicts:
#	spec/system/math_rendering_spec.rb
@HamptonMakes HamptonMakes changed the title Avoid stale math highlight reads in browser specs Synchronize browser specs with capybara-lockstep Nov 12, 2026
@HamptonMakes

Copy link
Copy Markdown
Collaborator Author

🤖 @codex review

@HamptonMakes

Copy link
Copy Markdown
Collaborator Author

🤖 Hampton requested the shared capybara-lockstep integration in place of the math spec's JavaScript snapshot. Published 5b31365d2a21080b089d3e96848ca0ca8d921823 implements it, including test-only tracking for received Turbo Stream renders, ActiveAdmin coverage, explicit manual mode for in-flight regressions, and a Ruby assertion that retains ordered exact highlight text.

Merged current main (1d720fc4) and resolved the math assertion conflict without dropping the upstream intent: Capybara re-queries live marks, and exact count/text, source range, formula exclusion, and reload checks remain. Production navigation and visual styles are unchanged. The integration uses the gem's normal timeout logging/next-command synchronization; logged timeouts do not substitute for feature assertions.

Local evidence: 34 final merged-base examples passed; the delayed real-cable regression fails with the Turbo hook removed and passes when restored. Focused editor/review/reaction checks and changed Ruby lint also passed; the PR describes their scope and the earlier strict-timeout probes. No shared database was reset and no full local suite was run alongside other workers.

Current-head CI and a new Codex review are pending. The existing ready-for-review PR state is preserved, but engineering handoff awaits those results. The separate caret failure remains unexplained; historical cancelled coverage stays recorded. No merge is authorized.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: 5b31365d2a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@HamptonMakes

Copy link
Copy Markdown
Collaborator Author

🤖 @codex review

@HamptonMakes

Copy link
Copy Markdown
Collaborator Author

🤖 CI correction published as 7742ffe9bbcb7a86fb81b4447bc7f9fbff363edd:

The first full run failed two checkbox assertions in browser shard 2 at seed 49204. They held Selenium elements across the saved-version body replacement. Both stale-element failures reproduced locally. The assertions now query the current fields by task name; checked/unchecked state, saved content, revision, attribution, and provenance checks remain.

The complete checkbox file plus the shared synchronization regressions passed locally: 17 examples at seed 49204. Changed Ruby lint and diff checks passed. This was a test-selector correction; the Lockstep integration and product code are unchanged.

All five stress jobs, both non-browser database suites, browser shards 1 and 3, lint, and the external checks passed in the earlier run. Browser shard 4 was cancelled, so that run is not a full pass. The new head gets a fresh full run; no unchanged failed test was retried. A fresh Codex review was also requested on this head. The caret-position follow-up remains unexplained.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: 7742ffe9bb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@HamptonMakes

Copy link
Copy Markdown
Collaborator Author

🤖 @codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: e1a7c82a05

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@HamptonMakes

Copy link
Copy Markdown
Collaborator Author

🤖 Final cleanup summary for Hampton's requested shared browser synchronization, head e1a7c82a05f8e97dff4a5cda7a017fa07709108a:

  • Added test-only capybara-lockstep for browser requests/resources and server requests. Host, engine, and ActiveAdmin pages include the shared setup. Received Turbo Streams remain tracked through their final renderer, including cancellation and errors. Tests that deliberately inspect unfinished work explicitly pause automatic synchronization. The guide documents default timeout logging and the limits around timers and future broadcasts; logged timeouts are not readiness assertions.
  • Replaced the math spec's JavaScript snapshot with Ruby inside Capybara's retry loop. Ordered exact text/count, source-range, formula-exclusion, and reload checks remain. CI exposed two old checkbox handles and a split synthetic drag gesture. Current-field assertions and a same-turn drop/dragend sequence fix those assumptions without changing product behavior or weakening persisted-result checks.
  • All fifteen checks passed on this exact head: full current-head CI, including both database suites, all four browser shards, all five stress jobs, voice cleanup, uploads, lint, and external checks. The earlier failed runs remain recorded; no unchanged CI failure was rerun to obtain a pass.
  • Codex review completed with no findings on e1a7c82a05. Local changed-code review found no outstanding issues. Current main 1d720fc4 is incorporated, the working tree is clean, and GitHub reports no merge conflicts. The existing ready-for-review PR state is preserved. No merge is authorized or performed.
  • Focused final validation passed: 34 merged-base examples, 17 checkbox/synchronization examples at seed 49204, and all three spring-folder plus four synchronization examples at seed 20822. The real cable regression fails with its shared hook removed. The folder race did not reproduce in one isolated old-code run; its CI trace and screenshot show the move completed before the old-row dragend failed. Ruby lint and diff checks passed. Local full-suite repetition was avoided on the shared Mac; CI supplied full coverage.
  • The PR body and browser testing guide provide dedicated-database tryout commands and expected results. The integration test database was dropped; the isolated probe database remains available for the local tryout. Production navigation and visual styles are unchanged.

The separate #311 caret-position failure remains unexplained. This integration does not retroactively complete its original cancelled shards, explain the obsolete custom runner's eight historical failures, or prove those findings fixed. Their evidence and limits remain visible.

@HamptonMakes
HamptonMakes merged commit f632b32 into main Nov 13, 2026
15 checks passed
@HamptonMakes

Copy link
Copy Markdown
Collaborator Author

🤖 Post-merge completion verified:

Release-note candidate:

Developer tooling: Coordinate browser tests with requests and received Turbo updates using capybara-lockstep. Keep feature assertions readable in Ruby, support tests that deliberately inspect unfinished work, and avoid stale checkbox references and split synthetic drag gestures. Contributed by Hampton Lintorn-Catlin (@HamptonMakes), with implementation assistance from Codex; built on Makandra's capybara-lockstep. (#326)

This records the merged benefit and contributor credit; no release was published by this cleanup. The change does not claim to eliminate every timing-sensitive failure. The separate #311 caret-position finding remains unexplained, and historical cancelled coverage and the obsolete runner's eight failures retain their documented limits.

Hampton's merge acceptance completes this PR's handoff; another tryout is not a gate. Existing checkout, preview, and data are preserved. Archiving this worker chat after recording this evidence; no further implementation or release action is pending here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant