From c07c4184e94d7be52b32e40be79bbe2ac26e978a Mon Sep 17 00:00:00 2001 From: Olivier Biot Date: Wed, 7 Oct 2026 19:29:42 +0800 Subject: [PATCH 1/2] Spatial audio: a movable listener, sounds placed in world pixels Positional audio could only be reached by doing the work the engine should be doing. The listener was welded to the origin with no public call to move it, so a game had to subtract the player's position from every source, every frame, and re-express it in the listener's axes by hand. The panner defaults were WebAudio's, which are metres, so a sound given pixel coordinates went near silent a few tiles out. And the two ways to place a sound quietly excluded each other. Added: - `follow`, `at` and `stopWithTarget` on `audio.play()`. A sound tracks a renderable's absolute position after every world update, or pins to a fixed point, and can end with the entity that owns it. `audio.unfollow` detaches one and leaves it playing. - `audio.setListener(target)`, `audio.listener()` and `audio.listenerOrientation()`. A `Camera3d` target contributes its basis as well as its position. Source positions become absolute world coordinates rather than offsets from the player. - `audio.setSpatialDefaults()` / `getSpatialDefaults()`, starting from `refDistance: 240`, `maxDistance: 10000` and `equalpower`. Everything takes melonJS world coordinates, y measured down; the Y flip into Web Audio's Y-up space happens in exactly one place. Fixed, all found while building the above: - `stereo()` and `position()` drive one panner node per voice, and the first call fixed which kind it was, making the other a silent no-op for that voice's life. The node follows the last writer now. - `audio.panner(name, attrs)` with no id never wrote the clip's own defaults, so a call before the first `play()` was lost entirely and every later voice inherited construction-time values. It writes the group, and merges rather than replaces. - Sound effects kept playing while the window was away, because pausing the game only ever reached the music track. The mix is muted on blur and restored on focus, gated by the existing `pauseOnBlur` and `stopOnBlur` settings rather than a new one. Opt-in end to end: until a game asks for a listener or a placement, no frame handler is installed and every existing call behaves as before. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NGvtaUNATVCVxD2qcbiY4t --- packages/melonjs/CHANGELOG.md | 10 +- .../melonjs/skills/melonjs-audio/SKILL.md | 120 ++++- .../melonjs/src/application/application.ts | 11 + packages/melonjs/src/audio/audio.ts | 15 + packages/melonjs/src/audio/autopause.ts | 96 ++++ packages/melonjs/src/audio/backend/spatial.ts | 146 ++++-- packages/melonjs/src/audio/playback.ts | 35 ++ packages/melonjs/src/audio/spatial.ts | 494 ++++++++++++++++++ packages/melonjs/src/audio/state.ts | 66 +++ packages/melonjs/src/audio/types.ts | 46 +- packages/melonjs/src/index.ts | 5 + packages/melonjs/tests/audio-spatial.spec.js | 488 +++++++++++++++++ 12 files changed, 1447 insertions(+), 85 deletions(-) create mode 100644 packages/melonjs/src/audio/autopause.ts create mode 100644 packages/melonjs/src/audio/spatial.ts create mode 100644 packages/melonjs/tests/audio-spatial.spec.js diff --git a/packages/melonjs/CHANGELOG.md b/packages/melonjs/CHANGELOG.md index 08587b2f42..4ff76b22a7 100644 --- a/packages/melonjs/CHANGELOG.md +++ b/packages/melonjs/CHANGELOG.md @@ -2,9 +2,17 @@ ## [20.9.0] (melonJS 2) - _unreleased_ +### Added +- **Spatial audio placed in world coordinates**: `audio.play(name, { follow: renderable })` tracks a sound to a renderable every frame, `{ at: { x, y } }` pins one to a fixed world point, and `{ stopWithTarget: true }` ends it when that renderable is destroyed. The numbers are world pixels with y measured down, the same ones already in `pos`, so a game converts nothing by hand. `audio.unfollow(id)` detaches a sound and leaves it playing where it is +- **A movable listener**: `audio.setListener(target)` puts the ear on a renderable and keeps it there, while `audio.listener(x, y, z)` and `audio.listenerOrientation(fx, fy, fz, ux, uy, uz)` place it by hand and read it back when called with no arguments. Source positions become absolute world coordinates instead of offsets from the player, and a `Camera3d` target contributes orientation as well, taken from its own basis. It is opt-in end to end: until a game asks for a listener no frame handler is installed, and every existing `position` / `stereo` / `panner` call behaves exactly as it did +- **Panner defaults shaped for pixels**: a placed sound now starts from `refDistance: 240`, `maxDistance: 10000`, `distanceModel: "inverse"` and `panningModel: "equalpower"`, so it carries across a screen instead of going near silent a few tiles out. `audio.setSpatialDefaults()` and `audio.getSpatialDefaults()` change what new voices inherit + ### Fixed +- Audio: `stereo()` and `position()` drive the same per-instance panner node, and the first of the two to be called decided what kind of node it was, which made the other a silent no-op for the life of that voice. The node now follows whichever was called last, so the order no longer matters +- Audio: `audio.panner(name, attributes)` with no playback id wrote only the voices already playing and never the clip's own defaults, so a call before the first `play()` changed nothing at all, the getter read back construction-time values forever, and every later voice inherited those. It writes the group now, and merges rather than replaces, so setting one attribute leaves the others alone +- Audio: sound effects kept playing while the window did not have focus, because pausing the game only ever reached the current music track. A looping effect such as an engine hum or an alarm played on over whatever the player had switched to. The mix is muted on blur and restored on focus, which covers `tone()` and `noise()` and anything a game hangs off `getMasterGain()` as well as ordinary clips, and it is gated by the existing `pauseOnBlur` and `stopOnBlur` settings so a game that deliberately keeps running in the background keeps its audio too - Skills: the UI skill listed `NineSliceSprite` panels among what it covers but never mentioned them. It now shows both ways to make one (`createSpriteFromName(name, { width, height }, true)` from an atlas, or the constructor from an image) and the three things that trip people up: `width` and `height` are the size to stretch to and are mandatory, `insetx` / `insety` default to a quarter of the image rather than its border, and the anchor is the centre -- Skills: the audio skill now covers what positional sound needs and did not say: the listener is fixed at the origin with no public call to move it, so positions are relative to the player; the panner defaults assume metres, so a sound placed in pixels is near silent a few tiles out (`refDistance: 1` puts one 100 px away at about 1%); and `stereo()` and `position()` drive the same per-instance panner, so whichever runs first makes the other a silent no-op. It also shows how to put reverb, echo or a compressor on the whole mix through `getMasterGain()`, noting that a streamed clip bypasses it, and documents `hasAudio()`, `hasFormat()`, runtime `audio.load()`, and the `rate` and `fade` ranges +- Skills: the audio skill now covers what positional sound needs and did not say. It documents the movable listener and the `follow` / `at` play options, the pixel-shaped panner defaults and how to widen them, and that every one of these calls takes world coordinates with y measured down. It also shows how to put reverb, echo or a compressor on the whole mix through `getMasterGain()`, noting that a streamed clip bypasses it, and documents `hasAudio()`, `hasFormat()`, runtime `audio.load()`, and the `rate` and `fade` ranges ## [20.8.0] (melonJS 2) - _2026-10-06_ diff --git a/packages/melonjs/skills/melonjs-audio/SKILL.md b/packages/melonjs/skills/melonjs-audio/SKILL.md index 0e98eaf0f8..7a74d66b77 100644 --- a/packages/melonjs/skills/melonjs-audio/SKILL.md +++ b/packages/melonjs/skills/melonjs-audio/SKILL.md @@ -76,8 +76,9 @@ audio.play("sfx", { sprite: "jump" }); The optional third element marks a region as looping. `play()`'s second argument takes either the original `loop` boolean or a `PlayOptions` object -(`sprite`, `loop`, `onend`, `volume`) — both forms work. Object fields win over -the positional `onend` / `volume` arguments when both are supplied. +(`sprite`, `loop`, `onend`, `volume`, plus the spatial `follow`, `at` and +`stopWithTarget`) — both forms work. Object fields win over the positional +`onend` / `volume` arguments when both are supplied. ## Clip state @@ -111,6 +112,14 @@ audio.muted(); // global mute state `audio.unload(name)` / `audio.unloadAll()` free the decoded buffers; `unload` also clears the current-track pointer if it named that clip. +### Losing focus mutes everything + +When the window loses focus the engine mutes all audio, and unmutes it on +focus. That covers sound effects, procedural `tone` / `noise` and the mixer +effects, not just the music track that `state.pause()` pauses. Set +`pauseOnBlur: false` in the application settings if a clip has to keep playing +in the background. + ## Spatial audio ```js @@ -126,44 +135,95 @@ Called with just the clip name, each of the four returns the current value — `stereo` gives `0` and `position` gives `[0, 0, 0]` before they have ever been set. -### The listener is fixed at the origin +### Let the engine place the sound: `follow` and `at` + +The shortest path to positional audio is a play option. Both take **world +pixels**, the same numbers as `pos.x` / `pos.y`, and the engine converts to +audio space for you: + +```js +// tracks the renderable every frame, and stops when it is destroyed +audio.play("ufo", { follow: this, stopWithTarget: true }); + +// pinned to a fixed world point +audio.play("portal", { at: { x: 1200, y: 480 } }); +``` + +`follow` re-reads the target's **absolute** position each frame, so a renderable +nested in a moving container is placed correctly. `stopWithTarget` is what you +want for a sound that belongs to an entity: without it a looping clip on a +destroyed renderable keeps playing from wherever it last was. `audio.unfollow(id)` +detaches a sound and leaves it playing where it is. -The ear sits at `(0, 0, 0)` facing `-z`, and **there is no public call to move -it**. Positions are therefore *relative to the listener*: for a sound to come -from the left of the player, pass the source's position minus the player's, -every frame either one moves. +### Move the listener, don't do the subtraction yourself ```js -// in the emitter's update() -audio.position("engine", this.pos.x - player.pos.x, 0, -0.5, this.engineId); +audio.setListener(player); // tracked every frame +audio.listener(x, y, z); // or place it by hand +audio.listener(); // → [x, y, z] in world pixels +audio.listenerOrientation(fx, fy, fz, ux, uy, uz); +audio.setListener(null); // stop tracking ``` -### Distances are in the units you pass — and the defaults assume metres +With a listener in the world, source positions are **absolute world +coordinates**, not offsets. The older idiom — passing `source.pos.x - +player.pos.x` every frame with the ear stuck at the origin — still works, but +only if you leave the listener at `(0, 0, 0)`; mixing the two double-counts the +player's position. -The panner defaults are `distanceModel: "inverse"`, `refDistance: 1`, -`rolloffFactor: 1`, `maxDistance: 10000`, `panningModel: "HRTF"`. With -`refDistance: 1`, a source 100 units away plays at about **1%** volume, so a -sound positioned in raw pixels is effectively silent a few tiles out. Either -set `refDistance` to the distance in pixels at which it should still be at full -volume, or scale positions down (divide by a tile size) before passing them: +`setListener` places the ear at the target's absolute position after the world +updates. It does not touch orientation, so a side-scroller keeps the default +"facing into the screen" and gets left-right panning plus distance falloff with +one call. + +**Y is down.** Every one of these calls takes and returns melonJS world +coordinates, where y grows downward; the Y-up flip Web Audio wants happens +inside the engine. Pass `pos.y` as it is. + +### The defaults are tuned for pixels + +`audio.setSpatialDefaults()` / `getSpatialDefaults()` set the panner attributes +every new voice starts from, and the shipped defaults already assume a 2D game +measured in pixels: + +| attribute | default | why | +|---|---|---| +| `refDistance` | `240` | full volume out to roughly a few tiles | +| `maxDistance` | `10000` | inaudible past ~2000 px with the inverse model | +| `panningModel` | `"equalpower"` | cheap, and it does not smear a 2D mix | +| `distanceModel` | `"inverse"` | smooth rolloff | + +So `audio.play("ufo", { follow: ufo })` is audible at a sane range with no +tuning. Widen or tighten the field once, globally: ```js -audio.panner("engine", { refDistance: 200, rolloffFactor: 1 }); -// or: audio.position("engine", dx / 32, 0, -0.5); +audio.setSpatialDefaults({ refDistance: 400, maxDistance: 20000 }); ``` -### `stereo` and `position` are one or the other +`audio.panner(name, attrs, id?)` still overrides per clip or per voice, and it +merges over the current values rather than replacing them, so writing one field +no longer resets the rest. Called without an `id` it writes the **group** +defaults, which is what a clip-wide tuning call looks like. + +For a 3D scene where front-to-back matters, `panningModel: "HRTF"` is worth the +cost; for a 2D one it mostly muddies the stereo image. + +### `stereo` and `position`: last writer wins + +Both drive the same per-instance panner node, and the engine swaps the node to +match whichever you called **last**: `stereo()` after `position()` gives you +plain left-right panning, and `position()` after `stereo()` gives you 3D +falloff. Neither silently does nothing any more, and the order you call them in +no longer matters. -Both drive the same per-instance panner node, and **the first one called -decides what that node is**: a stereo panner after `stereo()`, a 3D panner after -`position()`. After that the other call silently does nothing on that instance. -For a 2D game, left-right `stereo()` is usually all you want; reach for -`position()` only when distance falloff matters, and then use it alone. +Still pick one per voice and stay with it. A 2D game usually wants `stereo()` +alone; reach for `position()` (or `follow` / `at`) when distance falloff +matters. **Spatial audio is WebAudio-only.** On a clip loaded with `stream: true` (or `html5: true`) `stereo` / `position` / `orientation` return early and do nothing — which is exactly the case for the long music tracks people reach for -first. +first. `follow` and `at` are equally inert on a streamed clip. ## Procedural sound @@ -327,10 +387,14 @@ either way — blank screen, empty console. | a streamed clip ignores `withCredentials` | `stream: true` plays through an `