Add live device screen streaming to the Screen Capture window by todi1856 · Pull Request #224 · Unity-Technologies/com.unity.mobile.logcat

todi1856 · 2026-09-23T12:51:29Z

https://jira.unity3d.com/browse/PLAT-22403

Type of change:

Description

The main logcat window is not affected by this change, only screen capture window is affected

How to test? Go to https://yamato.ds.unity3d.com/project/277/branch/livestream/jobDefinition/.yamato%2Fwrench%2Fpackage-pack-jobs.yml%23package_pack_-_mobile_android-logcat/recent-jobs , take latest artifacts, extract them, and add com.unity.mobile.android-logcat-1.5.0.tgz package to Unity. tgz file will have the necessary jar inside.

Video

You can now watch the device's screen live in the Screen Capture window, and click, scroll and type into it from the Editor. The window opens from Window > Analysis > Android Screen Capture, so it no longer has to be reached through the Logcat window.

Capturing needs something on the device: External/UnityLogcatServer is a small Gradle project that builds an 18 KB jar, run through app_process as the shell user. It mirrors a display into an ImageReader, JPEG-encodes each frame and writes it over an abstract socket that adb forward exposes. Frames are only produced when the screen changes, so an idle device costs almost nothing. The jar is a build output rather than a committed binary, and a Yamato job builds it before the package is packed — that half is also up on its own as #226, cherry-picked from this branch to make it reviewable.

Input travels back up the same socket: touch, scroll wheel, keys through the device's own keyboard layout, and Ctrl/Cmd + A, C, V for select-all/copy/paste on the device. Back / Home / Overview buttons sit beside the image, which is the only way in on gesture-navigation devices, and below them Device Rotation (Auto, 0°, 90°, 180°, 270°) and, on a foldable, Device Fold (Auto, Fold, Half, Unfold) drive the device into a state without touching it. Take Screenshot is there too, so a screenshot is taken from where you are watching, at full device resolution whatever the stream is scaled to.

The window was reorganised around it. Screenshots used to overwrite one file in Temp; they are now kept, listed down the left under a Live toggle button, and each one is saved with a .json details file beside it recording the device, its Android version and the display size at the time. Where they go is a setting — Captures Folder, Library/AndroidLogcat/Screenshots by default.

The list takes a selection of more than one row (Shift and Ctrl click, Ctrl+A), and arrow keys, Home and End move through it. F2 renames, Del deletes, and right-click offers Show in Explorer / Reveal in Finder, Open, Copy To..., Rename, Delete and Select All, with the multi-row entries saying how many they will act on. Double-click opens a capture in whatever the OS associates with it. Ctrl+Shift+S (Cmd+Shift+S on macOS) takes a screenshot while the live view is showing.

A status bar along the bottom says where the last capture was written and what the stream is doing — the device, the display size, what it is scaled to and the frame rate cap. Stream size, JPEG quality and frame rate cap, and the captures folder, are in Preferences > Analysis > Android Logcat Settings, reachable from the window's own toolbar menu.

Improved screenshot window

Settings

Checklist for PR maker

Have you added a backport label? (new feature, no backport intended — confirm)
Have you updated the Changelog?
Have you added or updated the Documentation to your PR?

Testing status

Existing or new automation tests - what automation was added, changed
New tests are covering
- start/stop live steam
- frame delivery
- touch, keys
- scroll
- restart
- streaming against a device that was asleep
- screenshot list, delete, rename and external-change tests.

Devices:

Pixel 2 (Android 10)
Pixel 9 Pro Fold (Android 16)
- Folding/Unfolding while viewing
CPH1831 (Android 8.1).

Testing checklist

Built and run editor Locally or Yamato
Run on device Android/iOS (if applicable)
All items have tooltips?

A small Java server, built with Gradle into a dex jar and pushed to the device, that mirrors a display into an ImageReader, encodes each frame as JPEG and writes it to an abstract unix socket. It also reads control messages back up the same socket and injects touch, scroll, key and text events. The jar is a build output and is not committed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The window now lists saved screenshots beside a Live row that mirrors the device. Selecting Live starts the server on the device, shows its frames, and forwards touch, scroll, keyboard and text back to it. Along with it: a shared zoom and pan viewer for the live view and screenshot previews, a details file written beside each capture, a stats column, device rotation, and settings for the stream's size, quality and frame rate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Integration tests covering the stream end to end: frames arriving, injected touch, scroll, keys and text taking effect on the device, streaming a device that was asleep, and an unknown display id. Edit mode coverage for the details file written beside a screenshot. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The jar the live stream pushes to the device is a build output and is not committed, so a clone has none. A build_server_jar job produces it and the pack job now depends on it, which is also what puts it in the tarball the test jobs install. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

todi1856 · 2026-09-23T12:51:42Z

/review

u-pr · 2026-09-23T12:51:49Z

ℹ️ Skipped — a harness review for this commit (5aced5d) already ran or is in flight. Same-commit re-runs are paused for 10 min; push a new commit, or re-comment /harness-review in ~10 min to force a fresh one.

_{🤖 Helpful? 👍/👎}

u-pr

💡 Harness Review

The streaming pipeline is thoughtfully integrated, but several reachable UI, persistence, and startup edge cases can leave the displayed or saved content inconsistent or prevent streaming.

_{Reviewed commit 5aced5d}

_{🤖 Helpful? 👍/👎}

todi1856 · 2026-09-23T13:21:02Z

/review

u-pr

💡 Harness Review

The new capture workflow has state-management issues when device selection changes or a live-stream reader does not terminate promptly.

_{Reviewed commit 6508758}

_{🤖 Helpful? 👍/👎}

todi1856 · 2026-09-23T13:36:39Z

/review

u-pr

💡 Harness Review

The new live-view and persistent screenshot workflow has correctness gaps on reachable edge paths.

_{Reviewed commit f1d921f}

_{🤖 Helpful? 👍/👎}

todi1856 · 2026-09-23T13:48:18Z

/review

u-pr

💡 Harness Review

The live-stream feature is broadly cohesive, but keyboard shortcut forwarding can emit an unmatched key-down event for a normal release order.

_{Reviewed commit c58dc0f}

_{🤖 Helpful? 👍/👎}

todi1856 · 2026-09-23T14:29:11Z

/review

u-pr

💡 Harness Review

The streaming and screenshot-management implementation is broadly cohesive, but saving metadata-less screenshots can preserve incorrect provenance.

_{Reviewed commit b5bf917}

_{🤖 Helpful? 👍/👎}

todi1856 · 2026-09-24T08:29:07Z

/design-consistency

todi1856 · 2026-09-29T12:28:29Z

/review

u-pr · 2026-09-29T12:29:16Z

⚠️ Review skipped — this PR is too large to process.

The diff contains approximately 133,820 tokens (~401,461 characters), which exceeds the maximum limit of 120,000 tokens.

Please consider splitting this PR into smaller, focused changes.

_{🤖 Helpful? 👍/👎}

todi1856 · 2026-10-02T07:54:01Z

/review

u-pr

💡 Harness Review

The new live view and screenshot list add useful capture workflows, but their selection and input cleanup paths can leave the window acting on the wrong capture or holding mouse input after a stream failure.

_{Reviewed commit 24e1104}

_{🤖 Helpful? 👍/👎}

u-pr · 2026-10-02T07:57:14Z

💡 Harness Review

The new live view and screenshot list add useful capture workflows, but their selection and input cleanup paths can leave the window acting on the wrong capture or holding mouse input after a stream failure.

_{Reviewed commit 24e1104}

_{🤖 Helpful? 👍/👎}

todi1856 · 2026-10-02T10:38:43Z

/review

u-pr

💡 Harness Review

The change adds live screen streaming and a saved screenshot list; the review found new cases where screenshot cleanup or metadata handling can delete user files, and where changing the captures folder can show a capture from the old folder.

_{Reviewed commit 5fd1532}

_{🤖 Helpful? 👍/👎}

u-pr · 2026-10-02T10:41:19Z

+
+        // Start from an empty folder. Leftovers from an earlier run are still listed,
+        // and they carry another device's prefix or a name a test is about to reuse.
+        foreach (var screenshot in Runtime.CaptureScreenshot.GetScreenshots().ToArray())


Running a screenshot integration test calls Cleanup in both setup and teardown. It uses the real runtime's configured captures folder, then deletes every listed screenshot, including captures the test did not create and screenshots from other devices. Use a dedicated test folder and remove only files created by the fixture.

_{🤖 Helpful? 👍/👎}

@u-pr fixed

Looks good. Redirecting CaptureOutputDirectory to a dedicated folder under the test artifacts path in [OneTimeSetUp] and restoring it in [OneTimeTearDown] ensures test runs isolate their screenshots and avoid deleting captures from other devices or previous user runs.

_{🤖 Helpful? 👍/👎}

todi1856 · 2026-10-02T10:57:00Z

/review

u-pr

💡 Harness Review

The change adds live device viewing and a persistent screenshot list; the list’s refresh logic can leave its selection and preview out of sync after external file changes.

_{Reviewed commit 00478fc}

_{⚠️ Earlier blocking findings with unresolved threads: 1. The verdict above does not cover them; resolve each thread after addressing it.}

_{🤖 Helpful? 👍/👎}

todi1856 and others added 5 commits September 23, 2026 15:40

Document the live stream and update the changelog

5aced5d

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

u-pr Bot reviewed Sep 23, 2026

View reviewed changes

todi1856 added 5 commits September 23, 2026 16:06

fix preview going out of bounds

5db42b1

handle virtualDisplay being null

f87a4dd

Fix save as for screenshot

7123543

shorten changelog

076c7de

Try to fix panning on OSX

6508758

u-pr Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread com.unity.mobile.android-logcat/Editor/AndroidLogcatScreenCaptureWindow.cs Outdated

Comment thread com.unity.mobile.android-logcat/Editor/AndroidLogcatLiveStream.cs

todi1856 added 3 commits September 23, 2026 16:35

update screenshot

b2e6c2f

apply review fix

58fd75e

apply review fix

f1d921f

u-pr Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread com.unity.mobile.android-logcat/Editor/AndroidLogcatCaptureScreenshot.cs

Comment thread com.unity.mobile.android-logcat/Editor/AndroidLogcatLiveStream.cs

Apply review fixes

c58dc0f

u-pr Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread com.unity.mobile.android-logcat/Editor/AndroidLogcatLiveStream.cs

Track keys which were held

b5bf917

u-pr Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread com.unity.mobile.android-logcat/Editor/AndroidLogcatScreenshotInfo.cs

todi1856 added 3 commits September 23, 2026 17:36

show live view when opening the window

68f3993

Decouple layout viewer from screen capture

638bbef

LayoutView->LayoutViewer

a280502

todi1856 added 5 commits September 28, 2026 19:20

Add fold control

fbed84a

move access to preferences to kebab icon

47c3c2a

Make a help box

32b0d14

folder to path

fa949b5

bump image

48b19ef

todi1856 mentioned this pull request Sep 30, 2026

Livestream server #226

Merged

todi1856 added 4 commits October 1, 2026 18:01

apply changes from upstream

119aa80

update test

1158bdb

apply review

ccc14e9

Merge remote-tracking branch 'origin/master' into livestream

24e1104

u-pr Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread com.unity.mobile.android-logcat/Editor/AndroidLogcatScreenCaptureWindow.cs Outdated

Comment thread com.unity.mobile.android-logcat/Editor/AndroidLogcatLiveStream.cs

u-pr Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread com.unity.mobile.android-logcat/Editor/AndroidLogcatScreenCaptureWindow.cs Outdated

Comment thread com.unity.mobile.android-logcat/Editor/AndroidLogcatLiveStream.cs

todi1856 added 5 commits October 2, 2026 11:01

fix formatting

a938216

make screenshot shortcuts only work while in Live

ebd6a4f

flash screenshot when it's taken

2a7b665

release touch, if stream dies

18d47fc

apply cleanup

5fd1532

u-pr Bot reviewed Oct 2, 2026

View reviewed changes

todi1856 added 3 commits October 2, 2026 13:45

Screenshot->Capture

92c86d6

use dedicate folder for tests for captures

3f5c187

Ensure we're saving to correct folder

00478fc

u-pr Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread com.unity.mobile.android-logcat/Editor/AndroidLogcatCaptureList.cs

todi1856 changed the title ~~WIP: Add live device screen streaming to the Screen Capture window v2~~ Add live device screen streaming to the Screen Capture window Oct 2, 2026

todi1856 requested a review from BorisShir October 2, 2026 11:07

Conversation

todi1856 commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Type of change:

Description

Improved screenshot window

Settings

Checklist for PR maker

Testing status

Devices:

Testing checklist

Uh oh!

todi1856 commented Sep 23, 2026

Uh oh!

u-pr Bot commented Sep 23, 2026

Uh oh!

u-pr Bot left a comment

Choose a reason for hiding this comment

💡 Harness Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

todi1856 commented Sep 23, 2026

Uh oh!

u-pr Bot left a comment

Choose a reason for hiding this comment

💡 Harness Review

Uh oh!

Uh oh!

Uh oh!

todi1856 commented Sep 23, 2026

Uh oh!

u-pr Bot left a comment

Choose a reason for hiding this comment

💡 Harness Review

Uh oh!

Uh oh!

Uh oh!

todi1856 commented Sep 23, 2026

Uh oh!

u-pr Bot left a comment

Choose a reason for hiding this comment

💡 Harness Review

Uh oh!

Uh oh!

todi1856 commented Sep 23, 2026

Uh oh!

u-pr Bot left a comment

Choose a reason for hiding this comment

💡 Harness Review

Uh oh!

Uh oh!

todi1856 commented Sep 24, 2026

Uh oh!

todi1856 commented Sep 29, 2026

Uh oh!

u-pr Bot commented Sep 29, 2026

Uh oh!

todi1856 commented Oct 2, 2026

Uh oh!

u-pr Bot left a comment

Choose a reason for hiding this comment

💡 Harness Review

Uh oh!

Uh oh!

Uh oh!

u-pr Bot commented Oct 2, 2026

💡 Harness Review

Uh oh!

Uh oh!

Uh oh!

todi1856 commented Oct 2, 2026

Uh oh!

u-pr Bot left a comment

Choose a reason for hiding this comment

💡 Harness Review

Uh oh!

todi1856 commented Sep 23, 2026 •

edited

Loading