Fix TilemapGPULayer bleeding neighbouring tiles on flipped tiles

[moufmouf]

This PR (delete as applicable)

• Fixes a bug

This issue happens ONLY when using TilemapGPULayer on flipped assets. When using flipped assets, TilemapGPULayer can sample a texel that lies outside the tile's own frame in the tileset, drawing a one-pixel line of an unrelated tile along tile edges: horizontal lines for flipY, vertical ones for flipX. This does not happen if the tile is not flipped in the Tiled map.

This PR fixes this bug.

I opened an example here: phaserjs/examples#411

Before (the bleeding can appear only briefly when loading the page):

(see the colored lines?)

After:

Why? (disclaimer: the explanation below is by Claude):

getLayerData() builds the in-tile sampling coordinate with fract(), so uv spans a half-open range and the frame's far edge is never addressed — sampling stays inside the frame by construction. The flip flags then mirror it:

if (flipX) { uv.x = 1.0 - uv.x; }
if (flipY) { uv.y = 1.0 - uv.y; }

which turns that half-open range into a closed one. A fragment landing exactly on a tile boundary, where fract() is 0, now yields uv at the far edge on the flipped axis, and frameCorner + tile.uv addresses frameCorner ± tileSize: the first texel of the neighbouring frame in the atlas. Under NEAREST that is a whole row or column of an unrelated tile.

Two things make it conspicuous rather than a stray pixel. A whole screen row shares one interpolated outTexCoord, so the bleed is a full-width line; and the tile grid is regular, so when the camera's sub-pixel offset lines tile boundaries up with fragment centres, every boundary bleeds at once. The result is a grid of lines over the layer that appears and disappears as the camera scrolls — which is how this was first reported to us, on a map where players walking around made lines flicker in and out every 32px.

Unflipped tiles are unaffected, since uv never reaches the far edge. The CPU TilemapLayer is correct throughout: it emits one quad per tile with per-tile UVs.

The fix: inset the sample by half a texel, which is what the existing FEATURE_BORDERFILTER branch already does for LINEAR filtering. Under NEAREST this cannot move a sample out of the texel it already lands in, so it is a no-op for every fragment except the out-of-frame one. The BORDERFILTER path is deliberately left alone: it measures the overshoot itself in order to blend with the neighbouring tile, and clamps separately.

Verified two ways.

A shader-level harness compiles TilemapGPULayer.frag as shipped (pragmas stripped) and renders a synthetic layer through a quad offset half a pixel, so boundaries land on fragment centres:

stock 4.2.x                             patched
=== default (NEAREST path) ===          === default (NEAREST path) ===
tile 0, no flip        pass             tile 0, no flip        pass
tile 3, flipY          FAIL             tile 3, flipY          pass
   bleeding rows: 0,32,64,96,...
tile 2, flipX          FAIL             tile 2, flipX          pass
   bleeding cols: 0,32,64,96,...
tile 3, flipX + flipY  FAIL             tile 3, flipX + flipY  pass

=== FEATURE_BORDERFILTER ===            === FEATURE_BORDERFILTER ===
all four               pass             all four               pass

Through the engine, the example below flips one tile four ways and sweeps 512 sub-pixel camera offsets: 6 bleeding rows (the flipY quadrants) and 8 bleeding columns (the flipX quadrants) before, none of either after.

Note that whether a given fragment lands exactly on a boundary depends on how the driver interpolates the varying, so the triggering sub-pixel phase is machine-specific — but the out-of-frame address is in the shader either way.