Skip to content

feat(provider-webos): add LG webOS TV provider plugin - #4098

Merged
thymikee merged 1 commit into
callstack:mainfrom
alimek:feat/provider-webos
Nov 2, 2026
Merged

thymikee merged 1 commit into
callstack:mainfrom
alimek:feat/provider-webos

Conversation

@alimek

@alimek alimek commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds @agent-device/webos, an optional provider plugin that drives web apps on LG webOS TVs, and the private @agent-device/tv-cdp package it bundles, which the Tizen plugin in #4168 shares. The TV enters as a plugin-owned Android TV device (webos:<ares device>): lifecycle through ares-launch, the page over CDP through the ares-inspect tunnel.

Scope is the smaller v1 from review: open/close/apps, snapshot (-s, -d, -i), remote keys (directions, select, back), pointer, text, scroll, screenshot, logs, appstate. SSAP system buttons, perf memory, network dump, install/uninstall and the capability-matrix column follow in separate PRs.

agent-device plugins add @agent-device/webos
agent-device connect webos --device living-room
agent-device open com.example.app
agent-device snapshot -i

52 files. Also: kernel cdp-dom producer, an interactive-test option in capture-kit's desktop shaper, smoke fixtures, docs, a one-line provider-plugin catalog in help so help remote keeps its first-screen rules as plugins are added, and the TV flows of help remote in their own module (cli-help-tv.ts), which keeps cli-help.ts within its line limit. No linked issue.

Validation

Tested SHA 70420cb73, rebased on main bb960da.

  • pnpm check:affected --run: every stage passed except unit, whose failures are unrelated. fill-verification and committed-source-tree also fail intermittently on main. Two script tests failed under machine load and pass when run alone.
  • pnpm check:coverage-changed at a9301f326, before the rebase: changed-line coverage is 370/408 (90.69%) against the 70% threshold. pnpm check:coverage-crap reports no touched function at or above 30.
  • Unit tests cover the device route against fakes: the ares-inspect tunnel, connectTv, cancellable open, a lease that ends while open waits for the page, exit-only aresError rewrites, empty fill, pointer, scroll, screenshot, covered controls, Map/Set log previews, and the inspector error in the open timeout.
  • Live at b942ccd96: fill "" clears a filled input, and open --relaunch keeps one tunnel. pgrep -fl ares-inspect prints nothing after disconnect and after Ctrl-C during open.
  • Live at a9301f326: snapshots hit-test interactive nodes. A control under another element is marked [covered], press refuses it with the cover named, and snapshot -i still lists it. The output is in the PR conversation.
  • An independent review found four issues, all fixed.
  • Live on an LG 32LQ63006LA (webOS TV 22, SDK 7.6.0, firmware 04.60.91): connect, open, snapshot (41 nodes) / -i (18) / -s <label> (that subtree only), keys, screenshot, logs, appstate, apps, close. After disconnect, pgrep -fl ares-inspect prints nothing. An app closed outside agent-device gives the typed "No webOS app is attached" error in 0.15 s; with the TV unreachable, open returns COMMAND_FAILED in 2.9 s; without the CLI, TOOL_MISSING.

Risks: experimental; no CI device lane.

🤖 Generated with Claude Code

@vercel

vercel Bot commented Oct 30, 2026

Copy link
Copy Markdown

@alimek is attempting to deploy a commit to the Callstack Team on Vercel.

A member of the Team first needs to authorize it.

@alimek
alimek marked this pull request as ready for review October 30, 2026 14:00
@alimek
alimek marked this pull request as draft October 30, 2026 14:01

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 41 files

Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

View guided diff | Re-trigger cubic

Comment thread packages/provider-webos/src/runtime.ts Resolved
Comment thread packages/tv-cdp/src/cdp.ts Resolved
Comment thread packages/provider-webos/src/ssap.ts Resolved
Comment thread packages/provider-webos/src/index.ts Resolved
Comment thread packages/provider-webos/src/poll.ts Resolved
Comment thread packages/provider-webos/src/poll.ts Resolved
Comment thread packages/provider-webos/src/ares.ts Resolved
Comment thread website/docs/docs/webos.md Resolved
@alimek
alimek force-pushed the feat/provider-webos branch 2 times, most recently from 75ba3f6 to 1cd5272 Compare October 30, 2026 15:06
@thymikee

Copy link
Copy Markdown
Member

Thanks for the webOS provider. At 1cd5272 I found three defects in the lease and snapshot paths, and the PR has no live device run yet. Two fixes are needed in code before this can merge, and then a hardware run. The only failing check is Vercel, which is waiting for a team member to authorize the deployment of a fork commit, so no build ran. This does not look related to the code.

release(lease) only deletes the lease id (runtime.ts:235). The WebosSession in tvs keeps its ares-inspect child, the CDP page socket and the page log buffers. Only shutdown calls detachApp. After disconnect or idle expiry, the ssh tunnel and DevTools client stay attached until the daemon stops. This blocks other inspector clients, and the next lease on that TV inherits the previous page and logs. The rule should be: everything a lease opens is closed when that lease is released, and release and shutdown share one teardown. When no other lease holds the TV, release should run detachApp and drop the session. Then the "outliving the leases" comment and the README sentence can go. The 60 s idle expiry is best handled at the lease TTL owner, not by keeping the tunnel alive past release.

When the page socket closes (app exits, tunnel drops, TV sleeps), onclose only calls replies.failAll (cdp.ts:73). session.page and session.inspector stay set, and nothing watches the ares-inspect child after attach (ares.ts:69-75). So openCurrent and reconnect still hand out a log reader for a dead page, and every command fails with the DevTools hint instead of "No webOS app is attached. Run open first." Is it possible to make session.page valid only while its socket is open? For example, attachApp could register a close hook that runs detachApp when the page is still current, and the same could happen when the inspector child exits. The existing page() guard would then give the typed error.

snapshot() takes no options and returns the whole DOM walk on the android channel (interactor.ts:211). Capture-kit skips the post-wire scope pass for android, because android producers scope inside their own projection (snapshot-state.ts:173-180). So snapshot -s <label|@ref> returns the unscoped tree, and the output still reads as scoped. I did not trace whether -d and the interactive-only filter have the same gap. Every producer should either scope in its own projection or sit on a channel that gets the post-wire pass. The smallest fix is to return shapeDesktopSurfaceSnapshot(result, options) from @agent-device/capture-kit/snapshot-desktop-projection, which already applies scope, interactive-only and depth. Please add a test that a scoped capture through the provider snapshot binding returns only the subtree.

This is a new device-facing provider with no CI device lane, and the PR body names no TV or webOS version and shows no command output. I could not run a device, so everything above comes from reading the code. At the fixed head, on a named LG TV and webOS version (Settings > General > About), please paste: (1) connect webos --device <name> then open <appId>; (2) snapshot -s <label> showing only that subtree; (3) disconnect, then pgrep -fl ares-inspect printing nothing; (4) with the TV off or unreachable, open <appId> returning a JSON error with code COMMAND_FAILED (or TOOL_MISSING without the CLI) within the ares timeout, with no hang.

Could the lease own the session? allocate would create the WebosSession, release would run detachApp and delete it, and shutdown would loop over release. DOM captures could go through shapeDesktopSurfaceSnapshot, or onto the web channel so scope, depth and screenshot overlays use the web rules. For the web channel, please first check whether daemon admission accepts a web snapshot backend on an android platform device. If it does not, we need an ADR on non-mobile TV devices first. A smaller v1 would be lifecycle, CDP page, snapshot, input, screenshot and logs, at about half of the roughly 1,716 net production lines. SSAP pairing and buttons, perf memory, network dump, install/uninstall and the capability-matrix column could follow in separate PRs.

Not blocking, take or leave: runtime.ts and interactor.ts have no tests, so a runtime test with the scripted ares bin from seams.ts could cover DEVICE_IN_USE, release killing the inspector, and the scoped snapshot; cdp.ts is a second CDP client beside InspectorSession in the react-native package, so lift a shared one into host-kit when a third user appears; webosLease in scripts/capability-matrix/collect.ts builds the lease with as unknown as DeviceLease, so build a full lease or reuse the other columns' builder; and website/docs/docs/webos.md:95 says expiry ends the session while the README says the page stays attached, so after the release fix state one behavior and track the TTL need as a plugin-declared lease TTL issue.

On the earlier inline threads: the SSAP TLS certificate check still applies as a non-blocking note. The claim is right, but LG TVs serve self-signed certificates and the dispatcher is scoped to SSAP sockets, so one line in the README limitations is enough. These no longer apply and can be resolved: discussion_r4424516045, r4424516057, r4424516090, r4424516102, r4424516113, r4424516119, r4424516127, r4424516145, r4424516148, r4424516158, r4424516165, r4424516171, r4424516178, r4424516187, r4424516194, r4424516200, r4424516228, r4424516237, r4424516248, r4424516274, r4424516281, r4424516288, r4424516299, r4424516306, r4424516318 and r4424516335 are fixed at this head. r4424516260 does not apply because the webOS fixture has no environment, so the CLI cannot reach the server.

@alimek
alimek force-pushed the feat/provider-webos branch from 1cd5272 to 3209d66 Compare October 30, 2026 19:23
@alimek

alimek commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review. I took the smaller v1 and fixed the three defects. The PR is now one commit at 3209d66a7.

Scope. v1 covers lifecycle, the CDP page, snapshot, input, screenshot, logs and appstate. SSAP pairing and buttons, perf memory, network dump, install/uninstall and the capability-matrix column will follow in separate PRs. The CDP layer now lives in a private @agent-device/tv-cdp package, which a Tizen plugin stacked on this PR will reuse.

Lease teardown. The lease owns the session. release detaches the page, kills ares-inspect, closes the CDP socket and drops the session with its logs. shutdown runs the same teardown for every lease. A lease released while open is still running closes the tunnel that open creates. The "outliving the leases" comment is gone, and the README and webos.md now describe the same expiry behavior.

Dead pages. A closed page socket or an exited tunnel detaches the page, so later commands get "No webOS app is attached. Run open first." On the TV, an app that exits leaves the tunnel and socket open, and calls just time out. The plugin therefore also polls the tunnel's /json target list every second while a page is attached, and detaches when the page's target is gone.

Scope, depth, interactive-only. snapshot returns shapeDesktopSurfaceSnapshot(result, options). Every DOM node carries a rect, which the default interactive test counts as interactive. So the shaper now takes an optional interactive test, and the CDP producer passes hittable, editable or focused. runtime.test.ts drives the provider snapshot binding against a loopback CDP server. It covers a scoped capture returning only the subtree, -i dropping text-only nodes, DEVICE_IN_USE, release closing the tunnel, and detach on a socket close, a tunnel exit and an app exit.

Web channel. I kept the android channel with the shaper rather than checking web admission on an android device. I can raise that as an ADR question separately if you prefer it.

Evidence on an LG 32LQ63006LA (webOS TV 22, SDK 7.6.0, firmware 04.60.91). Labels in the outputs come from a private app, so I've put counts here instead.

  1. connect webos --device <name>, then open <appId>: Opened: <appId>.
  2. snapshot -s <label>: 13 of 37 nodes, only that container and its children. -s @e1 gives the same result. snapshot -i gives 18 of 41 nodes.
  3. disconnect, then pgrep -fl ares-inspect: no output, exit 1.
  4. With the TV unreachable, open <appId> --json returns COMMAND_FAILED ("ares-launch ERR! [Tips]: Connection timed out…") in 2.9 s. Without the CLI on PATH it returns TOOL_MISSING in 0.3 s.
  5. An app closed outside agent-device (ares-launch --close) gives the typed error in 0.15 s. Before the fix it waited for the 20 s DevTools timeout.

Non-blocking items.

  • The webosLease cast went away with the matrix column.
  • The SSAP TLS note will be part of the SSAP PR.
  • tv-cdp is now the shared TV client. I agree that it, together with InspectorSession, should move into host-kit once a third user appears.
  • Dropping SSAP also dropped undici. CDP now uses Node's built-in WebSocket, which shrinks the plugin bundle from about 520 kB to 68 kB.

@alimek
alimek marked this pull request as ready for review October 30, 2026 19:50

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 45 files

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

Comment thread packages/tv-cdp/src/runtime.ts Resolved
Comment thread packages/tv-cdp/src/snapshot.ts Resolved
Comment thread packages/provider-webos/src/ares.ts Resolved
Comment thread packages/tv-cdp/src/page-interactor.ts Resolved
Comment thread packages/tv-cdp/src/replies.test.ts Resolved
Comment thread packages/tv-cdp/src/index.ts Resolved
Comment thread website/docs/docs/webos.md Resolved
Comment thread packages/tv-cdp/src/page-log.ts Resolved
Comment thread packages/provider-webos/src/ares.ts Resolved
Comment thread packages/provider-webos/src/ares.ts Resolved
@thymikee

Copy link
Copy Markdown
Member

The earlier review at 1cd5272 asked for live evidence and several fixes. At 3209d66 the code has no defect that stops a merge: the author's summary from an LG 32LQ63006LA (webOS TV 22) covers the live evidence, and the Cubic threads on the DEVICE_IN_USE refusal, the socket open timeout, scroll amounts, ssap removal, reply failure handling, and the other items marked fixed at this head are addressed. I did not run the TV myself, and I did not rerun pnpm check:affected or the unit suite.

Not blocking, take or leave: in attachPage (https://github.com/callstack/agent-device/blob/3209d66/packages/tv-cdp/src/page-interactor.ts#L49) session.page is stored before the readiness poll but session.tunnel only after it. If the lease ends during the poll, teardown closes the page but finds no tunnel, so the poll spins to its 30 s deadline and open fails with "did not finish loading" instead of the lease-ended error. The rule is that everything open creates is reachable from the session before its first await, so one teardown closes it; set session.tunnel and session.appId before connectCdp, and end the poll when session.page !== page. I traced this from the code and did not reproduce it. A release-during-readiness case next to runtime.test.ts:227 would cover it.

One design question: does v1 need the separate @agent-device/tv-cdp package, with its TvPlatform/TvConnection seam, tvPluginConnection, TV_PLUGIN_BUNDLE, async lazy index, and the extra manifest, tsconfig, layering and Fallow entries? Its only consumer is provider-webos, and no Tizen PR exists yet. Could the CDP runtime stay as modules inside packages/provider-webos, calling ares through host-kit runCmd, and be extracted in the Tizen PR when a second consumer shows what is shared? Nothing has to change first for the smaller shape.

Of the other open threads, these still apply: the hit test for hittable (#4098 (comment)), the cancelled open polling (#4098 (comment)), and the unverified empty insertText (#4098 (comment), which needs a TV check); six lower-priority threads also still apply. These do not apply and can be resolved: ssap.ts is removed from v1 (#4098 (comment)), and the concurrent allocate check has no await between the holder check and leases.set (#4098 (comment)).

The only failing check is Vercel, and it shows the fork-deployment authorize link for 3209d66, so no build ran. The diff touches the website route that Vercel builds (website/astro.config.mjs, webos.md, commands.md, plugins.md), so the docs build is unproven. There are no conflicts. Before merge, please decide the package question and have a team member authorize the Vercel deployment so the docs build runs.

@thymikee

thymikee commented Nov 2, 2026

Copy link
Copy Markdown
Member

This conflicts with main now. Please rebase it so review can continue.

@alimek

alimek commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review. The branch is now at 19e6d4422, rebased on main (a0e7c7dc2).

attachPage order: fixed. open now stores the tunnel and app id on the session before connectCdp, and the readiness poll stops once the session no longer holds that tunnel. A lease that ends during the poll now fails open at once with the lease-ended error, and the tunnel is closed. The new test a lease released while the page loads ends open with the lease error and no tunnel sits next to the in-flight release test; on the old code it times out.

tv-cdp as its own package: I'd like to keep it. The Tizen plugin is in #4168 (draft, stacked on this branch). Its own commit adds packages/provider-tizen, about 350 lines of source for sdb lifecycle, keys and runtime wiring, and changes no file in packages/tv-cdp. Folding tv-cdp into provider-webos now would mean moving it back out in #4168. If you still prefer the smaller shape for v1, I'll fold it in.

Threads at the new head:

  • Cancelled open: fixed. The signal now reaches ares-launch, the page wait and the readiness poll, and a cancelled open closes its tunnel.
  • Empty insertText: fill with an empty value now selects the field's contents and presses Backspace. Unit tests cover it; it is not yet checked on the TV.
  • hittable hit test: left open as deferred. It changes snapshot -i output on TV apps, so I'd rather do it in a follow-up with a live run.
  • The six lower-priority threads have replies, and the two you marked as not applying are resolved.

Vercel: a team member needs to authorize the fork deployment; I can't start it from the fork.

@thymikee

thymikee commented Nov 2, 2026

Copy link
Copy Markdown
Member

Thanks for the update. The conflict from the earlier review is gone, and the review fixes now in 19e6d44 are in place: cancel handling in the inspect poll, the empty-value fill, the docs TTL steps, the preview log, and the host-kit command runner. One item still needs proof from the TV.

The change moves every ares call to host-kit runCmd/runCmdBackground (ares.ts:49). ares-inspect is now a background child that close() kills, and an empty fill now sends a text-less Backspace keyDown after select(). The PR body says these fixes were not re-run on the TV, and the earlier pgrep evidence came from the old spawn path. A regression here would show only on the device. Please run this at 19e6d44 or later and paste the output: connect webos --device <name>, open <app>, snapshot -i showing a text field with a non-empty value, fill <ref> "", then snapshot -i (or get) showing that value empty. Then open <app> --relaunch, apps, close, disconnect, and finally pgrep -fl ares-inspect printing nothing. Optionally Ctrl-C an open mid-load and show pgrep is empty too.

Not blocking, take or leave: aresError rewrites more than exit failures, so a cancelled or timed-out open --relaunch becomes ares-launch failed and loses its typed reason and timeoutMs (ares.ts:138); the rule should be that it rewrites only an exit failure (check details.exitCode) and passes every other error through, with a webOS-level cancelled-open test asserting the code and reason. The cancelled-open test in runtime.test.ts:204 uses a bare assert.rejects, so it should assert isRequestCanceledError. No test calls fill at all, so the empty-fill Backspace branch (page-interactor.ts:221) is untested; a fill('') case against fakePage that records the Backspace keyDown and no Input.insertText would cover it, and the PR body claim that unit tests cover it should be corrected.

Of the open inline threads, the P2 on hittable in snapshot.ts (#4098 (comment)) still stands, and one lower-priority thread still applies. These can be resolved: ssap.ts is not in v1, so the TLS note returns with the SSAP PR (#4098 (comment)); the allocate race has no await between the holder check and leases.set (#4098 (comment)); and the cancel, fake-timer, docs TTL, preview log, inspector error and host-kit threads are fixed at this head (r4427923841, r4427923859, r4427923877, r4427923890, r4427923900, r4427923908). The empty-fill thread (r4427923851) is fixed in code but waits on the TV check and a test.

CI shows no result for this head: the only failing check is Vercel, which is the fork authorization link, so no build ran. The diff touches the website docs it builds, and only 2 checks are attached, so the repo gates (check:affected, unit, layering, fallow) did not run. I did not run the TV, pnpm check:affected or the unit suite, so the 14/15-stage local result is unverified, and I did not confirm that TV Chromium deletes a selection on a Backspace keyDown without text. Before merge, the TV run above is needed, and a team member must authorize the Vercel deployment and approve fork CI so the gates run on 19e6d44.

@alimek
alimek force-pushed the feat/provider-webos branch from 19e6d44 to b942ccd Compare November 2, 2026 13:07
@thymikee

thymikee commented Nov 2, 2026

Copy link
Copy Markdown
Member

The earlier request is partly fixed at b942ccd: the aresError guard is in. The device-facing route is otherwise unchanged since 19e6d44, and it has still not been run on the TV.

packages/provider-webos/src/ares.ts:49 still runs ares-* through host-kit runCmd/runCmdBackground, keeps ares-inspect as a background child that close() kills, and fills an empty value with select() plus a text-less Backspace keyDown. No new comment or PR-body update posts a run, and the earlier pgrep -fl ares-inspect evidence came from the old spawn path. So nothing shows that the tunnel still yields the localhost port, that it dies on disconnect, or that webOS Chromium deletes a selection on a text-less Backspace. Please run this on the LG TV at b942ccd or later and paste the output: connect webos --device <name>, open <app>, snapshot -i showing a text field with a non-empty value, fill <ref> "", then snapshot -i (or get) showing that field empty. Then run open <app> --relaunch, apps, close, disconnect, and finally pgrep -fl ares-inspect, which must print nothing. Optionally press Ctrl-C during an open while the page loads and show an empty pgrep again.

The Coverage check (run 41450332300) fails on lines this PR adds. Changed-line coverage is 246/401 = 61.35% against the 70% threshold, and the misses are in new files: the whole inspect tunnel in ares.ts (40, 51-109), connectTv in provider-webos runtime.ts, and the tv-cdp scroll, screenshot, hover, longPress, focus and type paths. The CRAP gate also flags four new functions at 0% coverage: page-interactor.ts scroll, the ares.ts:87 inspect probe, provider-webos connectTv and tv-cdp decode. Please fix this with tests, not exemptions. For example, add a fake ares-inspect script that prints localhost: for a fake /json server, plus scroll cases against fakePage. I did not run the new ares cancel test, the cancelled-open test or the empty-fill test; I only read the pre-fix code to confirm they would fail without the fix.

The open inline thread on tv-cdp snapshot.ts hittable still stands, with the author's follow-up plan: #4098 (comment). No conflicts are known. Vercel is still waiting for fork authorization, so the website docs build has not run and a team member needs to authorize it. Before merge, the live TV run and a passing Coverage check are needed.

@alimek
alimek force-pushed the feat/provider-webos branch from b942ccd to bbe772a Compare November 2, 2026 15:55
@alimek

alimek commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

Thanks. Both requests are done at bbe772aed.

Coverage. I added tests rather than exemptions.

  • ares.test.ts: a fake ares-inspect prints localhost:<port> for a fake /json server. The tests cover:
    • inspect returns the app page, and close() ends the tunnel process;
    • an early exit reports its output, and a missing CLI is TOOL_MISSING;
    • with no page in time, the error names the inspector's last failure and the tunnel is killed;
    • a cancelled inspect keeps isRequestCanceledError and kills the tunnel.
  • A new provider-webos/src/runtime.test.ts covers connectTv:
    • it leases the ares default TV, or the one --device names over the plugin option;
    • an unknown name is DEVICE_NOT_FOUND with the known TVs;
    • open --relaunch runs ares-launch --close, ares-launch, then ares-inspect on the leased TV;
    • close runs ares-launch --close.
  • runtime.test.ts in tv-cdp, against fakePage, covers:
    • scroll at the viewport centre and in a container;
    • screenshot, and the refusal of a hidden page;
    • tap, long press, focus, hover and type;
    • a remote key, and a missing key refused with the supported list;
    • listApps and appState through the platform binding, and its error once the lease is gone.
  • The tv-cdp app-log codec moved unchanged into app-log-descriptor.ts, which has a round-trip test and an invalid-input test for decode. This is the only production change in this round.

Local run at bbe772aed:

  • pnpm check:coverage-changed passes: changed-line coverage is 363/401 (90.52%), up from 61.35%, and branch coverage is 153/222.
  • pnpm check:coverage-crap reports no touched function at or above 30.
  • pnpm check:affected --run passes 14 of 15 stages. In unit, 17018 of 17020 tests pass; the two failures are fill-verification and committed-source-tree, which also fail intermittently on main.

Earlier review items, at b942ccd96:

  • aresError rewrites only a nonzero exit (host-kit's processExitError) and passes every other error through.
  • The tv-cdp cancelled-open test asserts isRequestCanceledError.
  • A fill('') test records the select, then keyDown/keyUp Backspace, and no Input.insertText.

TV run. Below is the run you asked for. It was made at b942ccd96. Since then, only tests and the codec move have changed, so the device route is the same code.

The CLI was built from the branch and the plugin installed from pnpm pack, on an LG 32LQ63006LA with webOS TV 22. Device name and app id are replaced, and output is trimmed to the relevant lines.

$ pgrep -fl ares-inspect
$ agent-device connect webos --device living-room
$ agent-device open com.example.app
Opened: com.example.app
$ agent-device snapshot -i          # on the app's search screen
Snapshot: 8 nodes
@e8 [input] "Search" [focused]
$ agent-device fill @e8 "fed"
Filled 3 chars
$ agent-device snapshot -i
Snapshot: 16 nodes                  # the search results appeared
@e8 [input] "Search" [focused]
$ agent-device get attrs @e8
  "identifier": "search-screen-input",
  "value": "fed",
$ agent-device fill @e8 ""
Filled 0 chars
$ agent-device snapshot -i
Snapshot: 8 nodes                   # the results are gone
@e8 [input] "Search" [focused]
$ agent-device get attrs @e8
  "identifier": "search-screen-input",
  "value": "",
$ agent-device open com.example.app --relaunch
$ pgrep -fl ares-inspect            # one tunnel, a new pid after the relaunch
98143 node .../bin/ares-inspect -d living-room -a com.example.app
$ agent-device apps
Ares Hosted App (com.sdk.ares.hostedapp)
Example (com.example.app)
$ agent-device close com.example.app
$ agent-device disconnect
Disconnected remote session "adc-ce98ff8581b7265b6b288d6e72b0fc5c".
$ pgrep -fl ares-inspect
$ echo $?
1

get attrs @ref reads the latest ref frame, so each read follows a fresh snapshot -i. A screenshot after fill @e8 "" shows the empty field with its placeholder.

Ctrl-C during open, then the same session afterwards:

$ agent-device connect webos --device living-room
$ agent-device open com.example.app &   # SIGINT after 3 s, before "Opened"
$ pgrep -fl ares-inspect
98843 node .../bin/ares-inspect -d living-room -a com.example.app
$ kill -INT %1                          # open exits 130 and prints nothing
$ pgrep -fl ares-inspect
$ echo $?
1
$ agent-device open com.example.app
Opened: com.example.app
$ agent-device appstate
Foreground app: com.example.app
$ agent-device close com.example.app
$ agent-device disconnect
$ pgrep -fl ares-inspect
$ echo $?
1

So TV Chromium does delete a selection on a text-less Backspace keyDown.

Open items:

  • The hittable thread stays open for the follow-up described there.
  • Vercel still needs a team member to authorize the fork deployment.
  • Fork CI needs approval so that Coverage and the repo gates run on bbe772aed.

@alimek
alimek force-pushed the feat/provider-webos branch from bbe772a to a9301f3 Compare November 2, 2026 16:21
@alimek

alimek commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

The hittable item is no longer open. It is fixed in a9301f326, and the thread is resolved.

  • The DOM walk hit-tests the centre of each interactive node with document.elementFromPoint. When the hit lands outside the node, the snapshot records the nearest named or interactive element there as the cover, or the bare fact of coverage when that element has no name.
  • tv-cdp publishes this through capture-kit's covered annotation. annotateSnapshotNodesCoveredByPolicy now also accepts the covering element; its existing caller is unchanged. The node gets hittable: false, interactionBlocked: 'covered', coveredBy and the [covered] hint, so click, press, fill and focus refuse it with the cover named.
  • snapshot -i keeps covered controls, because remote focus still reaches them.
  • Tests: the policy with a named cover and with an anonymous cover (capture-kit), and a covered control in a tv-cdp snapshot. The README and the website page describe [covered].

Live on the same LG webOS TV 22, with app labels shortened:

$ agent-device snapshot -i              # home screen, 16 nodes as before
@e8 [button] "<hero card 1>"
@e9 [button] "Watch" [covered]
@e10 [button] "<hero card 2>"
@e11 [button] "Watch" [covered]
$ agent-device press @e9
Error (COMMAND_FAILED): Ref @e9 is covered by another visible element and cannot be tapped safely
Hint: Covered by the button "<hero card 1>" at (80, 184 1163x675). Dismiss or close that element, scroll the target clear of it, or inspect with snapshot/screenshot before retrying.
  • Search, Shows, other screens: nothing is marked covered (8, 20 and 15 nodes), and fill on the search input still works.
  • Live screen: the focused card is marked covered, because its preview layer sits on top while it is focused. A coordinate tap at its centre did nothing, while a tap on an uncovered part of the card opened the player, so the mark is correct.

Local run at a9301f326:

  • pnpm check:coverage-changed passes at 370/408 (90.69%).
  • pnpm check:coverage-crap reports no touched function at or above 30.
  • pnpm check:affected --run passes every stage except unit, whose only failure is fill-verification, which also fails intermittently on main.

Still waiting on a team member: authorizing the Vercel deployment, and approving fork CI so Coverage runs on a9301f326.

@thymikee

thymikee commented Nov 2, 2026

Copy link
Copy Markdown
Member

Thanks for the update. The code looks good at a9301f3. The coverOf hit test now marks covered nodes and refuses press on the covered Watch button, so the earlier covered-element finding is fixed (#4098 (comment)), and that thread is already resolved.

Not blocking, take or leave: a text input whose centre sits under its own <label for> (a floating label without pointer-events:none) is marked [covered], because elementFromPoint returns the label and the ancestor walk names it as the cover. A real tap on that label focuses the input, so fill and press should not refuse it. The rule is that a hit the browser forwards to the node is not a cover, for example in coverOf return null when hit.closest('label')?.control === el (https://github.com/callstack/agent-device/blob/a9301f3/packages/tv-cdp/src/snapshot.ts#L51).

The coverage numbers and check:affected are your local runs, and the TV output is pasted from your runs. I did not run gates or drive a TV. The coverOf hit test has no unit test, because the runtime test writes covers by hand, so only the live run shows it. I also did not judge the timing of the new ares tunnel tests on Linux CI.

The only check that did not pass is Vercel, which is waiting for a team member to authorize the fork deployment. The diff touches the webOS docs pages it builds, so the docs build is unproven until then. Coverage, unit and layering have not run on this fork head. A maintainer must approve fork CI and authorize the Vercel deployment before merge.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Nov 2, 2026
Add @agent-device/webos, an optional provider plugin that drives web apps
on LG webOS TVs, and the private @agent-device/tv-cdp package it bundles,
which holds the CDP page layer a later Tizen plugin will share. The TV
enters as an Android TV device owned by the plugin (id webos:<ares
device>). Lifecycle runs through ares-launch, and the page is driven over
the Chrome DevTools Protocol through the ares-inspect tunnel.

v1 covers open, close, apps, snapshot, remote keys (directions, select,
back), pointer, text, scroll, screenshot, logs and appstate. Releasing a
lease closes the page and the tunnel, and a page whose socket, tunnel or
app goes away is detached, so later commands ask for open again.

- Kernel: add the cdp-dom snapshot producer on the android channel so DOM
  snapshots do not pose as UIAutomator output.
- Capture kit: let shapeDesktopSurfaceSnapshot take an interactive test,
  since every DOM node carries a rect.
- Packed-install smoke: let scripts/check-provider-plugin.mjs run fixtures
  for providers without credentials, and add the webOS fixture.
- Docs: add the LG webOS page, mention it in plugins, commands and the
  sidebar, and add a webOS flow to help remote.
@alimek
alimek force-pushed the feat/provider-webos branch from a9301f3 to 70420cb Compare November 2, 2026 18:20
@alimek

alimek commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on main bb960da51 at 70420cb73 to fix the Lint failure.

  • main brought cli-help.ts to 998 of its 1000 lines, so the merged file went over the limit. The TV flows of help remote now live in src/commands/schema/cli-help-tv.ts, and cli-help.ts is at exactly 1000 lines. feat(provider-tizen): add Samsung Tizen TV provider plugin #4168 adds its Tizen line to the new module.
  • main now requires a named session for every leased connection. The webOS flow in help remote, the README and the website page now export AGENT_DEVICE_SESSION=checkout-tv and explain REMOTE_SESSION_REQUIRED.
  • Local pnpm check:affected --run at 70420cb73 passes every stage except unit, whose failures are unrelated. fill-verification and committed-source-tree also fail intermittently on main. Under machine load, an Xcode isolation-scan test and an Android replay test also failed; both pass when run alone.

No device-facing code changed in this rebase.

@thymikee
thymikee merged commit 9d00ad0 into callstack:main Nov 2, 2026
16 of 18 checks passed
@thymikee

thymikee commented Nov 2, 2026

Copy link
Copy Markdown
Member

The code looks good at 70420cb. Coverage now passes, so the earlier coverage failure is resolved. No conflicts are known. Before merge, a maintainer must authorize the Vercel fork deployment so the webOS docs build can run, and the Apple Smoke Tests job needs a rerun.

Vercel reports “Authorization required to deploy” and never built; this diff changes the docs route it builds, including webos.md, so the rendered page remains unverified. Apple Smoke Tests failed in its iOS phase (macOS passed), and this diff changes no Apple provider or iOS runner code, so the failure looks unrelated; a rerun should confirm it.

I did not run gates or verify the reported local check:affected result or the claim that the unit failures are unrelated. There was no live TV run at this head; the b942ccd run still applies because the device route is unchanged from a9301f3. There are no open review threads.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants