Canvas: text metrics, multi-GUI filters, drawImage(canvas), toDataURL by bkaradzic-microsoft · Pull Request #1855 · BabylonJS/BabylonNative

bkaradzic-microsoft · 2026-08-27T23:19:19Z

Summary

Canvas2D text metrics, multi-GUI filter lifetime, Canvas-source drawImage, and PNG readback.

Area	Change
Text metrics	Em-scaled font sizes, advance and glyph ink bounds, and consistent no-font fallback properties. Ink bounds exclude SDF, interpolation-border, and blur padding.
Multi-GUI filters	Reference-count shared blur programs and uniforms across multiple ADTs.
`drawImage(canvas)`	Snapshot rendered source pixels with GPU readback. Apply floating-point source/destination rectangles, cropping, and negative-extent normalization to the GPU draw. Retain temporary uploads until queued NanoVG draws are flushed.
`toDataURL`	Encode GPU readback as PNG using `bimg::imageWritePng`; empty, case-variant, and unsupported media types fall back to PNG.
Regression coverage	Observe destination GPU pixels through a second canvas, rather than asserting only the CPU mirror. Regenerate the committed JavaScript test bundle.

Known visual-baseline differences

The existing native-canvas.png is a Native regression baseline, not browser parity. It also captures repeating rather than clamped gradient colors, a dark projected-texture perimeter, and Native-only Droid Sans input. These differences and the matched-input follow-up are documented in Polyfills/Canvas/Readme.md.

This review update does not regenerate golden images. Future golden changes must isolate the intended fix against Babylon.js with matching font bytes and scene inputs.

Validation of the review update

Windows x64 D3D11/Chakra UnitTests build succeeded.
CanvasTextMetrics.*:JavaScript.All: 2/2 GTests passed. JavaScript: 57 passing, 27 pending, no failures.
Five additional JavaScript.All repetitions passed with the same counts. The built JavaScript asset exactly matches the committed generated bundle.
Font-padding regression: 72 scenarios covering both coordinate origins, two sizes, three glyphs, and blur 0/4/20/40. Standalone SDF and non-SDF variants pass; the original implementation fails the regression.

The existing JavaScript webpack command omits the TypeScript Babel preset. For this bundle regeneration, the existing webpack config was used with the already-installed @babel/preset-typescript appended to its loader options in memory, without changing dependency manifests. Native execution used the resulting committed bundle.

Copilot

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated 4 comments.

Copilot

Pull request overview

Copilot reviewed 16 out of 18 changed files in this pull request and generated 4 comments.

bkaradzic-microsoft · 2026-09-01T23:10:17Z

CI was red across the board after merging master (#1857): Context.cpp still called the old 14-arg encoder->blit. Updated CaptureRGBA to the TextureRegion pair API (same pattern as NativeEngine/FrameBuffer).

Pushed in the latest commit.

Independent of instance-branding (BabylonJS#1844 / type tags): - fontstash: CSS/canvas font-size as em units; MeasureText advance + ink bounds - MeasureText binds the same face FillText uses - nanovg_filterstack: refcount shared blur programs (multi-ADT lifetime) - drawImage(canvas) via Canvas InstanceOf + CPU pixel mirror (no NativeInstanceRegistry) - toDataURL("image/png") via bimg_encode; link bimg_encode Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 60c2ec68-6de1-445d-9fc9-b699db737eae

GCC rejects default member initializers on a nested type used as an inline static member of the enclosing class. Match master: plain UniformHandle fields + value-init the static Uniforms instance. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 60c2ec68-6de1-445d-9fc9-b699db737eae

toDataURL and drawImage(canvas) previously read only the CPU pixel mirror (populated by putImageData/drawImage of bitmaps), so NanoVG draws such as fillRect/text/paths were missing from the result. CaptureRGBA now flushes pending NanoVG commands, blits the canvas RT to a READ_BACK texture, and blocks on DeviceContext::ReadTextureAsync. Mid-frame flushes complete readTexture requests so the wait works under an active FrameCompletionScope (RAF / unit-test open frame) without deadlocking. Also: - ForceMidFrameFlush API; process readTexture queue in PerformMidFrameViewFlush - Bilateral Canvas/Context detach on destroy (JS cycle finalizer order) - nvgCreateImageFromHandle helper for external bgfx textures (NODELETE) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 60c2ec68-6de1-445d-9fc9-b699db737eae

The fontstash em-square scale change makes canvas2d text match CSS px sizing (glyphs were previously ~0.86x too small for Droid Sans). Native Canvas playground (#TKVFSA#8) draws large stroked/filled text over shapes, so the old golden (captured at the undersized scale) fails pixel compare at ~4.1%. Same golden update as shotgun 520be83; CI rendered result is within 0.125% of this reference (limit 2.5%). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 60c2ec68-6de1-445d-9fc9-b699db737eae

- CaptureRGBA: unpremultiply GPU readback so toDataURL/getImageData/drawImage expose straight alpha (NanoVG RT is ONE/INV_SRC_ALPHA premultiplied). - MeasureText: use nvgTextBoundsInk so height/actualBoundingBox* are glyph ink extents; nvgTextBounds still uses the line box for layout callers. - FreeType fontstash path: FT_Set_Pixel_Sizes from em `size` directly so raster matches em-scaled advances (stb path already did). - Unit test: semi-transparent fill + getImageData channel asserts on drawImage(canvas) destination. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 60c2ec68-6de1-445d-9fc9-b699db737eae

Master BabylonJS#1857 changed encoder.blit to TextureRegion pairs; Context.cpp still used the old 14-arg form and broke CI after merging master. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d47cbab2-d751-4cf9-984f-4412dd9ec601

UBSan on MacOS_Sanitizers aborted during Canvas Context GC because nvgRenderDelete used `if (gl->prog.idx)` — idx 0 is valid and BGFX_INVALID_HANDLE is 65535, so never-created programs still called destroyProgram and indexed ProgramRef[65535]. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d47cbab2-d751-4cf9-984f-4412dd9ec601

memSet left prog.idx at 0, which bgfx::isValid treats as a real handle. Dispose then called destroyProgram(0) and asserted on Win32. Pair that with isValid for lazy program create on first flush (replacing the old !prog.idx gate). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d47cbab2-d751-4cf9-984f-4412dd9ec601

bghgary

[Reviewed by Copilot on behalf of @bghgary]

Concerns inline.

Hold the frame completion scope before NanoVG flush through GPU readback, and retain temporary image uploads until queued draws have been submitted. Apply floating-point source and destination rectangles, normalize negative extents, and proportionally clip source bounds while preserving draw state. Fall back to PNG for unsupported toDataURL types, complete fallback text metrics, and remove atlas border and blur padding from reported ink bounds. Add destination-GPU regressions and native font-metric coverage, regenerate the committed JavaScript bundle, and document existing golden divergences. Normalize the added C++ and test indentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 88569c10-a7ff-4373-9a58-afa9c68b8c09

Copilot

🟡 Changes recommended

Readback can block indefinitely, and several newly exposed text and image edge cases produce incorrect API results.

Get a fresh assessment by requesting another Copilot review.

Review details

Files reviewed: 21/23 changed files
Comments generated: 5
Review effort level: Balanced

Report unserviceable mid-frame flushes, keep pending readback resources alive, separate NanoVG image IDs from bgfx handles, prioritize Canvas branding, and preserve signed and zero-ink text bounds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 54aaa1e1-b4b3-46e7-9de4-5a56add4ac42

Keep NanoVG-created image IDs aligned with their existing bgfx handles while assigning externally registered handles IDs above the bgfx handle range. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 54aaa1e1-b4b3-46e7-9de4-5a56add4ac42

Expose the active bgfx renderer's GPU capability to JavaScript and mark only GPU-dependent Canvas regressions pending on no-op CI builds. Preserve real-GPU coverage, regenerate the test bundle, and document the configuration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 85ec8d5f-d470-4fcd-8a7e-40cc926df07d

Use the existing USE_NOOP_METAL_DEVICE test configuration to expose the Canvas GPU-test capability instead of including bgfx in Tests.JavaScript.cpp. The installation consumer only has public BabylonNative headers, so that include broke both the Windows and Linux installation jobs. Keep the pixel assertions enabled on Windows, Linux, and real Metal devices. Document the explicit no-op configuration rather than a runtime renderer query. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 48fa7ec8-0cf6-48a0-b836-10b7ace38c1a

bkaradzic-microsoft · 2026-09-15T17:52:38Z

Fixed the Windows and Linux installation failures in 2379eb3.

Both jobs failed while compiling Apps/UnitTests/Source/Tests.JavaScript.cpp: bgfx/bgfx.h is not part of the installed public headers. The previous no-op-renderer change added that private dependency to a source file also built by Install/Test.

The test host now uses the existing USE_NOOP_METAL_DEVICE definition, shared with the Apple test host, to set hasGpuRendering. Only the explicit no-op Metal configuration skips the GPU pixel cases; Windows, Linux, and real Metal configurations keep them enabled. No bgfx headers were added to the public package.

Verified the exact header regression locally: the previous source fails against installed headers, while the corrected source compiles with and without the no-op definition. Windows/QuickJS JavaScript.All also passes all 85 cases with the GPU assertions enabled. All 34 checks now pass on this exact commit, including both Windows and Linux installation jobs and all macOS configurations: completed CI run.

bghgary

[Reviewed by Copilot on behalf of @bghgary]
Concerns inline.

Read getImageData regions from the rendered framebuffer rather than the partial CPU mirror. Preserve straight alpha, transparent out-of-bounds pixels, resize behavior, and wide normalized region coordinates; surface readback errors as JavaScript exceptions. Validate drawImage arity and normalize geometry before source readback or uploads. Coerce coordinates once before observing source dimensions, and retain images referenced by queued draws through the NanoVG flush. Keep PNG encoding available independently of input-image loading and correct the imported texture-ID namespace comment. Replace probe-canvas assertions with direct getImageData coverage and add native no-op allocation/readback and PNG encoding regressions. Verify the native tests with image loading enabled and disabled. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 48fa7ec8-0cf6-48a0-b836-10b7ace38c1a

bghgary

[Reviewed by Copilot on behalf of @bghgary]
Concerns inline.

Allocate a missing render target even when the Canvas was never resized. Classify drawImage sources before numeric coercion, then read current dimensions and validate mutable ImageBitmap data before use. Cover both readback APIs on fresh canvases and source classification mutations during coordinate coercion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 48fa7ec8-0cf6-48a0-b836-10b7ace38c1a

Use the writer's output length instead of the memory block's rounded-up capacity. This prevents unused allocation bytes from being serialized and fixes the untouched Canvas PNG comparison on Windows and Linux CI. Add a deterministic regression checking that PNG chunks end exactly at IEND for multiple Canvas sizes, without weakening the readback assertions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 48fa7ec8-0cf6-48a0-b836-10b7ace38c1a

bghgary

[Reviewed by Copilot on behalf of @bghgary]
LGTM

Merge upstream/master 64f0683, including the Canvas feature and regression coverage from BabylonJS#1855. Reconcile upstream Canvas changes with the newer shotgun clipping, text layout, image lifetime, and PNG transparency fixes. Preserve the compute/prewarm flush budget while reporting whether forced flushes can complete, and complete pending readbacks during mid-frame flushes. Keep both upstream and shotgun test coverage and regenerate the bundle. Preserve integral browser font line-box metrics separately from precise glyph ink bounds. The coordinated Babylon.js change selects the font line box for Native GUI layout, avoiding regressions from the new API. Normalize the Canvas test font path and isolate no-font coverage before the combined suite registers a global font. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 60c2ec68-6de1-445d-9fc9-b699db737eae

…BabylonJS#1855) Canvas2D text metrics, multi-GUI filter lifetime, Canvas-source `drawImage`, and PNG readback. | Area | Change | |------|--------| | Text metrics | Em-scaled font sizes, advance and glyph ink bounds, and consistent no-font fallback properties. Ink bounds exclude SDF, interpolation-border, and blur padding. | | Multi-GUI filters | Reference-count shared blur programs and uniforms across multiple ADTs. | | `drawImage(canvas)` | Snapshot rendered source pixels with GPU readback. Apply floating-point source/destination rectangles, cropping, and negative-extent normalization to the GPU draw. Retain temporary uploads until queued NanoVG draws are flushed. | | `toDataURL` | Encode GPU readback as PNG using `bimg::imageWritePng`; empty, case-variant, and unsupported media types fall back to PNG. | | Regression coverage | Observe destination GPU pixels through a second canvas, rather than asserting only the CPU mirror. Regenerate the committed JavaScript test bundle. | The existing `native-canvas.png` is a Native regression baseline, not browser parity. It also captures repeating rather than clamped gradient colors, a dark projected-texture perimeter, and Native-only Droid Sans input. These differences and the matched-input follow-up are documented in [`Polyfills/Canvas/Readme.md`](https://github.com/bkaradzic-microsoft/BabylonNative/blob/0be41b62ad848d0f7df5102247ffa8800aa9c5cf/Polyfills/Canvas/Readme.md). This review update does not regenerate golden images. Future golden changes must isolate the intended fix against Babylon.js with matching font bytes and scene inputs. - Windows x64 D3D11/Chakra `UnitTests` build succeeded. - `CanvasTextMetrics.*:JavaScript.All`: 2/2 GTests passed. JavaScript: 57 passing, 27 pending, no failures. - Five additional `JavaScript.All` repetitions passed with the same counts. The built JavaScript asset exactly matches the committed generated bundle. - Font-padding regression: 72 scenarios covering both coordinate origins, two sizes, three glyphs, and blur 0/4/20/40. Standalone SDF and non-SDF variants pass; the original implementation fails the regression. The existing JavaScript webpack command omits the TypeScript Babel preset. For this bundle regeneration, the existing webpack config was used with the already-installed `@babel/preset-typescript` appended to its loader options in memory, without changing dependency manifests. Native execution used the resulting committed bundle. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Branimir Karadzic <branimirkaradzic@gmail.com> Co-authored-by: Branimir Karadzic <branimirkaradzic@Bgmail.com> Copilot-Session: 60c2ec68-6de1-445d-9fc9-b699db737eae Copilot-Session: d47cbab2-d751-4cf9-984f-4412dd9ec601 Copilot-Session: 88569c10-a7ff-4373-9a58-afa9c68b8c09 Copilot-Session: 54aaa1e1-b4b3-46e7-9de4-5a56add4ac42 Copilot-Session: 85ec8d5f-d470-4fcd-8a7e-40cc926df07d Copilot-Session: 48fa7ec8-0cf6-48a0-b836-10b7ace38c1a

## Summary - Intersect rectangular Canvas clips with the active NanoVG scissor so nested, disjoint, translated, signed-rectangle, and restored clips retain parent bounds. - Implement `clearRect` with opaque destination-out coverage, independent of `globalAlpha` and filters, while restoring the prior drawing state. - Initialize empty NanoVG filter stacks and add GPU-readback regressions with existing render-test skip handling. - Fix the OpenPBR validation scene's asynchronous mesh-loading race: return the scene only after mesh import and material setup finish, using the existing per-test source replacement support. The reference image, render count, and tolerance are unchanged. - Fix the inherited Vulkan CI gate to require both sampler-boundary tests to run and pass without assuming a fixed total suite size. ## Base Rebased onto `master` at `4af724a6`. Includes signed-rectangle normalization before scissor intersection. This includes the already-merged Canvas work from #1855 and does not depend on the open branding PR #1844. ## Validation - Windows x64 / D3D11 / QuickJS, `RelWithDebInfo`: all 6 selected native cases passed, including all 94 JavaScript cases. - A delayed mesh import reproduced every pixel of the blank CI screenshot on both the PR and its upstream base (41.223% mismatch). With the scene-loading fix, both delayed and ordinary runs pass at 0.733%, below the unchanged 2.5% limit. - The actual snippet's transformed scene creation waits for the import, finishes all 16 material assignments, and propagates import failures. - The Vulkan gate accepts the actual 34-case CI report and rejects missing, skipped, failed, unexecuted, or incomplete required boundary tests. ### Signed-rectangle update - Normalize negative width/height before intersecting with the parent clip; preserve the existing transform and antialiasing adjustment. - Cover both signed axes, their combination, and rotated nested clips using byte-identical GPU comparisons with equivalent positive rectangles. - Windows x64 / D3D11 / Chakra / RelWithDebInfo: all 6 selected native cases pass, including 68 JavaScript cases; 27 optional codec cases are pending because those plugins are disabled in this build. The new regression fails against the previous PR head on the first negative-width case and passes with the fix. - No Babylon.js changes, dependency updates, reference changes, or validation-gate adjustments in this update. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 60c2ec68-6de1-445d-9fc9-b699db737eae Copilot-Session: 48fa7ec8-0cf6-48a0-b836-10b7ace38c1a

Copilot AI lite review requested due to automatic review settings August 27, 2026 23:19

Copilot started reviewing on behalf of bkaradzic-microsoft August 27, 2026 23:35 View session

This comment was marked as outdated.

Sign in to view

bkaradzic-microsoft force-pushed the pr/canvas-features branch from 47eff94 to 384f8f8 Compare August 27, 2026 23:51

bkaradzic-microsoft mentioned this pull request Aug 27, 2026

Canvas: unforgeable native instance branding (Path2D/Gradient/Image) #1844

Draft

bkaradzic-microsoft force-pushed the pr/canvas-features branch from 384f8f8 to dd45487 Compare August 28, 2026 00:04

bkaradzic-microsoft requested a balanced review from Copilot August 28, 2026 00:19

Copilot started reviewing on behalf of bkaradzic-microsoft August 28, 2026 00:21 View session

Copilot AI reviewed Aug 28, 2026

View reviewed changes

Comment thread Polyfills/Canvas/Source/Canvas.cpp Outdated

Comment thread Polyfills/Canvas/Source/Context.cpp Outdated

Comment thread Polyfills/Canvas/Source/nanovg/fontstash.h Outdated

Comment thread Polyfills/Canvas/Source/Canvas.cpp Outdated

bkaradzic-microsoft requested review from CedricGuillemet and bghgary and a balanced review from Copilot August 28, 2026 03:01

Copilot started reviewing on behalf of bkaradzic-microsoft August 28, 2026 13:53 View session

Copilot AI reviewed Aug 28, 2026

View reviewed changes

Comment thread Polyfills/Canvas/Source/Context.cpp Outdated

Comment thread Polyfills/Canvas/Source/nanovg/fontstash.h

Comment thread Apps/UnitTests/JavaScript/src/tests.javaScript.all.ts Outdated

Comment thread Polyfills/Canvas/Source/MeasureText.cpp Outdated

bkaradzic-microsoft force-pushed the pr/canvas-features branch from 1b0a26d to a41eec2 Compare September 2, 2026 14:39

bkaradzic-microsoft mentioned this pull request Sep 11, 2026

Fix nested canvas clipping and clipped clears #1872

Merged

matthargett mentioned this pull request Sep 13, 2026

Roadmap: Canvas 2D / SVG conformance slices (dependencies, risk, test cross-references) rebeckerspecialties/femtovg#5

Open

bkaradzic-microsoft and others added 8 commits September 14, 2026 12:45

bkaradzic-microsoft force-pushed the pr/canvas-features branch from a41eec2 to 6283216 Compare September 14, 2026 20:22

bghgary requested changes Sep 14, 2026

View reviewed changes

bkaradzic-microsoft requested a review from bghgary September 14, 2026 22:21

bkaradzic-microsoft requested a balanced review from Copilot September 14, 2026 22:21

Copilot started reviewing on behalf of bkaradzic-microsoft September 14, 2026 22:26 View session

Copilot AI reviewed Sep 14, 2026

View reviewed changes

bkaradzic-microsoft and others added 4 commits September 14, 2026 16:11

bghgary requested changes Sep 15, 2026

View reviewed changes

Comment thread Polyfills/Canvas/Source/Context.cpp

Comment thread Polyfills/Canvas/Source/Context.cpp Outdated

Comment thread Polyfills/Canvas/Source/nanovg/nanovg_babylon.cpp Outdated

Comment thread Polyfills/Canvas/Source/Canvas.cpp Outdated

bkaradzic-microsoft requested a review from bghgary September 15, 2026 20:38

bghgary requested changes Sep 15, 2026

View reviewed changes

Comment thread Polyfills/Canvas/Source/Context.cpp

Comment thread Polyfills/Canvas/Source/Context.cpp Outdated

bkaradzic-microsoft requested a review from bghgary September 15, 2026 22:42

bghgary approved these changes Sep 16, 2026

View reviewed changes

bkaradzic-microsoft merged commit 64f0683 into BabylonJS:master Sep 16, 2026
65 of 66 checks passed

Conversation

bkaradzic-microsoft commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Known visual-baseline differences

Validation of the review update

Uh oh!

This comment was marked as outdated.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bkaradzic-microsoft commented Sep 1, 2026

Uh oh!

bghgary left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bkaradzic-microsoft commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bghgary left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bghgary left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bghgary left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bkaradzic-microsoft commented Aug 27, 2026 •

edited

Loading

bkaradzic-microsoft commented Sep 15, 2026 •

edited

Loading