Repository navigation
feat(game): game-services framework gaps and modules - #285
Draft
patrickleet wants to merge 40 commits into
Draft
patrickleet wants to merge 40 commits into
patrickleet wants to merge 40 commits into
Conversation
Extend the FactCommand deny list with amount, coins, gold, currency, price, cost, balance, grant, payout, credit and debit. The list moves to a shared `trusted` module so ticket extension claims can reuse it, and every denied field (Rust name, serde rename or alias) now gets its own spanned error instead of only the first. [[tasks/gr-p26-fact-economy-denylist]]
FactStamp gains emitted_at_ms (Clock::unix_ms, new required Clock method; ManualClock takes a Unix origin), activation (boot epoch) and config_version (HostConfig::config_version, settable from the host or a hook). Session vars and event metadata carry all six fields; cell::session_stamp and FactStamp::from_metadata read them back, and FactStamp::emission_order orders facts independent of commit order. Handoffs carry the source's departure time in the transfer ticket (`left`): source on_leave facts are pinned to it and the target stamps everything after it, so a join sorts after the leave even with clock skew. Replay format bumps to DGR3 (config version, departure time, SetConfigVersion event). [[tasks/gr-p25-fact-stamp-ordering]]
Tickets carry game-declared claims (match id, team, party) under `ext`. `#[derive(TicketExtension)]` declares them with the FactCommand deny list (shared module, JSON fixture checked by Rust and JS tests) and refuses rename_all/flatten. ServerConfig::with_ticket_extensions admits only the declared schema for join (any verifier) and transfer tickets; any ext without a schema is refused. Claims reach JoinRequest, PlayerInfo and ObserverInfo, are replaced on rejoin and carried by handoff transfer tickets. JS: mintGameTicket takes `extensions`, createGameTicketHandler gains a server-side `claims()` callback; the browser body still cannot add claims, and denied names or oversize objects answer 400. [[tasks/gr-p23-ticket-extension-claims]]
GameServer takes a ZoneFactory: ZoneFactory::new(|activation| async
{..}) may await storage, ZoneFactory::sync covers factories without
I/O. The factory now runs on the activation task after the ownership
claim and snapshot load and receives ZoneActivation { zone_id,
activation (fencing token), allocation }. Allocation records come from
a ZoneAllocationStore (with_allocation_store; memory store included).
Joins queued during creation are admitted afterwards; Ok(None) answers
NOT_FOUND, a factory or store error and a factory outliving
activation_timeout answer the new ZONE_UNAVAILABLE (9) code. Refusals
release the claim and are counted in activations_refused_total.
activate() now waits for the zone; begin_activation() only starts it.
Demos migrated.
[[tasks/gr-p22-zone-allocation-payload]]
With HostConfig::lifecycle_facts the host emits PlayerJoined (ticket,
rejoin, handoff), ConnectionLost (grace started), PlayerReconnected,
PlayerDisconnected (grace expired) and PlayerLeft { reason } as typed
FactCommand inputs bound to the verified principal. Keys use boot epoch,
player slot and a per-slot transition counter; a host backlog keeps them
in order instead of dropping them on backpressure, and they share the
principal's commit lane. New hooks Zone::on_connection_lost and
Zone::on_reconnected run at grace start and reconnect. Off by default.
Mob Pit enables them and projects presence (/dev/mobpit/presence),
ordered by fact stamp so a late handoff leave cannot mark a player
offline.
[[tasks/gr-p19-zone-lifecycle-facts]]
Zones declare `type Message` (a GameEvent-derived type, or NoMessages)
and handle Zone::on_message on their own task between ticks.
ZoneHost::deliver takes ZoneInbound::{Kick, Message}; the built-in kick
removes every connection of a principal with LeaveReason::Kicked (and a
PlayerLeft lifecycle fact).
GameServer::send delivers locally; POST /admin/zone-message (admin
secret) serves peers and services, answering 404 for inactive zones
(never activating them) and 421 wrong_owner with the owner's admin URL.
net::ZoneInboxClient resolves the owner via ZoneDirectory and follows
those answers. Deliveries are recorded and replayed. Messages appear in
the game surface (`messages`) and the generated TypeScript gains
MESSAGE, MESSAGES and encodeMessage.
[[tasks/gr-p21-zone-inbox]]
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Command handlers arm, re-arm and cancel cell alarms with their events (ctx.arm_alarm / ctx.cancel_alarm). Alarm rows live in the cell's own SQL (migration 11, cell_alarms) and commit in the command's transaction. When due, the cell dispatches the alarm's command into itself through its command ledger with a (name, generation) command id and a trusted cell-alarm session, so each arm has its effects at most once, even across a crash before settlement. On celld the one Durable Object alarm is shared with the outbox watchdog: every setAlarm targets the earliest of the next domain alarm and the next drain. AggregateCell::alarm(&env) fires due alarms, then drains. The in-process reference host and native cell-sqlite cells take a ManualClock (CellStreamStore::with_clock) and fire_due_alarms(). Proven on celld dev 0.4.0 and 0.6.2 by tests/celld/alarms.mjs (atomic arm, periodic re-arm, watchdog vs domain alarm, restart); CI runs it in the celld storage job. [[tasks/gr-p28-cell-alarms]]
SagaDefinition::action now returns SagaAction: a single command or a FanOut of up to MAX_SAGA_FAN_OUT (64) commands. Each member gets its own command id (type:saga:step:phase:attempt:member), saga-member metadata and timeout timer. The step succeeds when every member succeeds; it settles only after every member replied or timed out, then compensates only the members that applied or timed out, one at a time, highest index first. Late replies after a member timeout are ignored. compensation and on_step_succeeded take the member index. New events (member_succeeded, member_failed, fan_out_settled) replay to the same state. Breaking: action/compensation/on_step_succeeded signatures changed. [[tasks/gr-p27-saga-fanout]]
scripts/check-wasm32-domains.sh builds each domain crate's library for wasm32-unknown-unknown (e2e-ui todo/chat domains, the Mob Pit crawler, and game_services/crates/*-domain and kernel once they exist) plus distributed and distributed_game without features. A negative fixture with Tokio's net feature must fail, so the check provably bites. blob-domain is excluded with its reason: it mounts SQL read models. The integration-wasm32-domains workflow runs on PRs and gates the main release tag. [[tasks/gr-p30-wasm32-domain-check]]
LeaderboardSurface declares a Redis/in-memory board (or board family)
with per-role grants. GraphqlEngineBuilder::leaderboard serves its roots:
{name}Leaderboard(first), {name}LeaderboardAround(radius), {name}MyRank
and, only with rank_any_member, {name}Rank(member). Viewer roots bind
the member from the verified session claim, never an argument.
Top, around and my-rank are also @LiVe subscriptions, refreshed through
GraphqlEngine::leaderboard_change_listener registered on the
LeaderboardProjector (applied changes and promotions; duplicates are
silent). In protocol mode results carry an incomplete snapshot and live
frames use snapshot mode.
The declaration feeds the SDL, the authorization fingerprint, the client
manifest (optional leaderboards section, omitted when empty so other
fingerprints are unchanged) and the CLI's new leaderboards.ts.
CI runs the leaderboard conformance suite and the Redis-backed GraphQL
test against the game-demos Redis service.
[[tasks/gr-p29-leaderboard-graphql]] [[tasks/gr-p9-redis]]
One sink per zone server sends each fact to
POST {CELLD_URL}/{kind}/{shard}/{command}, routed by the fact's command
(by principal, by an input field, or lifecycle_route: zone.* facts go
to the account's presence cell, never to gameplay cells). Command ids
come from command_id_for(service, fact key), so a retry after a timeout
replays the cell ledger receipt. Principal and FactStamp travel as
x-user-id and x-game-* headers, only with the internal secret; cells
rebuild the session with cell::fact_session_from_headers.
celld unreachable, 5xx or in-progress is Retryable; with the new
RetryPolicy::until_settled facts stay queued (no local store) and the
zone sees Backpressure at capacity. Unrouted or rejected facts are
Permanent. The in-process CellFactSink is now native-only.
tests/celld_facts proves it on celld dev 0.4.0 and 0.6.2 with three
cell kinds, replay, the internal secret, and a celld crash/restart.
[[tasks/gr-p24-zone-celld-fact-routing]]
world::zone_lease adds the ZoneLease:{zone_id} aggregate: acquire
publishes zone.activated with a token that never repeats, renew
re-arms the expiry alarm with only an internal event (no outbox
rows), release publishes zone.released. The cell alarm publishes
zone.lease_expired { zone_id, holder, token } once a holder stops
renewing, re-checking the lease so a racing renewal never gives a
false expiry. celld::CellZoneOwnership implements ZoneOwnership over
these cells; world::ownership_conformance is the shared behaviour
suite (Local, FencedLease, ZoneLease cells).
Handlers read time from ctx.cell_now(): the cell host overwrites a
cell-clock claim on every dispatch, so callers cannot choose it.
Endless Grid and Ripple hosts use CellZoneOwnership with CELLD_URL
and DISTRIBUTED_INTERNAL_SECRET (default stays LocalOwnership); their
smoke tests pass on celld dev that way.
[[tasks/gr-p20-zone-lease-cell]]
The Mob Pit presence smoke test failed about half the time with a dropped player stuck Online. Root cause: FactStamp had no order inside one tick. A join and a dropped socket in the same tick and millisecond got equal stamps, so emission_order returned Equal and the presence projection dropped ConnectionLost as stale. - FactStamp gains `seq`, the host's per-activation emission sequence; every stamped fact takes the next number and emission_order compares it within one zone activation. It travels as `game.seq` metadata and the `x-game-seq` session variable (breaking: required on read). - The connection task now tells the zone when a socket write fails. Before, a write error skipped the Disconnect and the zone only found out if a later frame to that client failed to queue. - distributed_cli: the generation-rejecting readiness probe gets a 1500 ms budget; at 100 ms the initial probe timed out under a loaded workspace run and the supervisor exited before the first assertion. Refs [[tasks/gr-p19-zone-lifecycle-facts]]
Zones are aggregates, their players and mobs are entities, views are
read models and facts are domain events. Views stay projections, not
copies; only the hand-written replicate() loop goes.
- `#[replicated(.., of = Entity)]` with `#[from]` / `#[from(a.b)]`
fields generates `View::project(&entity, ViewRest { .. })`; when all
fields are `#[from]` it also implements `Project<View>`. A missing or
mistyped `#[from]` field is a compile error at the attribute
(trybuild). Views without `of` are unchanged.
- New `distributed_game::entity`: `EntityId`, `Entity`, `Project<V>`,
`EntityCollection`, `Entities<Z>` and `WorldView::project`.
- `Zone::entities()` registers collections once; the host projects them
each patch (`ZoneHost::world_view`), then calls `Zone::replicate`,
which now defaults to nothing. `WorldView::add` takes
`impl Into<EntityId>`.
- Mob Pit has no replicate(): players, mobs, pillars and the zone's own
HUD view are projected. A golden test runs a 900-tick fight and checks
every projected world and Delta frame byte-for-byte against the old
loop; generated TypeScript is unchanged (`run.sh gen --check`).
- README maps zone/entity/view/fact to aggregate/entity/read model/
domain event.
Refs [[tasks/gr-p16-game-entity]]
A command is now one aggregate method. Inside a #[sourced] impl,
`#[command("crawler.record_kill", shard = .., load = .., outcome = ..)]`
generates the portable mount (`pub struct RecordKill`), its handler and
the registration that portable_command! + a hand-written handler gave.
- The `#[principal]` parameter is mandatory and filled from the verified
session only; an input field named like it fails to compile (const
check over the new `CommandInputType::FIELD_NAMES`, filled by
`#[derive(CommandInput)]`).
- Input: one input parameter, or `input = T` to map the other params to
T's fields by name. load = required | or_create | create; shard = an
input field or the principal; outcome = succeeded | eventual (method's
Ok value is the payload) or Succeeded<P>/Eventual<P> + payload = f.
Optional roles, guard, field, transition, mount. Errors -> Rejected;
captured domain events are published with the commit.
- The domain_commands witness is used as the transition (also for
non-pub command methods).
- Receipt helper `Entity::last_recorded()` -> `Recorded { sequence,
timestamp }` + `unix_ms()`.
- Mob Pit `crawler.record_kill` migrated; its spec now also declares the
`crawler.mob_killed` emit set via `domain_commands::RecordKill`.
- Golden test: command specs, GraphQL SDL and client manifest equal the
hand-written registration. trybuild fixtures for the principal rules.
- The handler-fn `#[command(id = ..)]` names the sourced form when given
a string literal outside #[sourced]. README leads with the attribute.
Refs [[tasks/gr-p17-command-macro]]
Mob Pit's durable state moves to celld; the native `cell-sqlite`
backend is removed (decision: celld for local dev and deployment, no
native cell mode; the in-process reference host stays for unit tests).
- examples/game/cells: workers-rs worker with `crawler/{principal}`
(the #[command] RecordKill + CelldOutbox), `crawler-roster/all`
(celld cannot enumerate cells) and `presence/{principal}` (lifecycle
facts, stamp ordered). Its Queue consumer relays outbox messages to
the zone server's secret-protected /internal/celld-queue/relay, which
publishes them on the projection SQLite bus for mobpit-service.
- Zone server: MobPitFactSink over CelldFactSink (by_principal for
crawler.record_kill, lifecycle_route for presence, enlist before the
first kill), RetryPolicy::until_settled; profile, crawlers and board
rebuild read cells over the internal path. `--celld-url`/CELLD_URL and
DISTRIBUTED_INTERNAL_SECRET are required.
- game_demos::devcelld and client/tests/helpers/celld.mjs start a
private `celld dev` per test (CELLD_BIN); run.sh starts celld for
mobpit, leaderboard, sveltekit and smoke (0.6.2+ required, esbuild from
the client). CI installs celld v0.6.2 and worker-build.
BREAKING CHANGE: the `cell-sqlite` feature, CellSqlitePath,
AggregateCell::open_sqlite*, CellStreamStore::open_sqlite* and the native
SQL executor are gone. game_demos: CrawlerFactSink -> MobPitFactSink,
WorldOptions::new takes CelldOptions, spawn_with builds DemoRoutes.
[[tasks/gr-p32-mobpit-celld]] [[tasks/gr-p24-zone-celld-fact-routing]]
Zone hibernation snapshots move to celld and every demo with durable
state now runs on celld; SqliteZoneStateStore is removed (same decision
as cell-sqlite: no native durable store).
- world::zone_state: a ZoneState:{zone} aggregate cell (zone_state.save
/ zone_state.delete, token fencing equal to the old SQLite store; the
highest token survives a delete). Snapshots up to 64 KiB are inline;
larger ones go to the worker's R2 bucket binding (celld object storage,
RustFS in the Hops stack) with key/sha256/len in the cell; superseded
and refused uploads are deleted.
- celld::CellZoneStateStore client; world::zone_state::celld_worker
route glue (feature workers-rs). celld's R2 binding is not an instance
of R2Bucket, so the glue takes it unchecked (Env::bucket refuses it).
- tests/celld/fact-worker and examples/game/cells mount ZoneLease and
ZoneState cells; the demo worker also gets The Freeze's gem-pouch and
content-log cells.
- Grid and Ripple require celld: CellZoneOwnership + CellZoneStateStore
(no LocalOwnership/SQLite fallback); The Freeze settles its facts in
celld via FreezeFactSink. run.sh starts celld for freeze, grid and
ripple; Rust and JS cluster/freeze tests start their own celld dev.
- Hibernation tests (distributed_game world, celld_zone_state,
grid_smoke) run against celld; CI's storage job runs them with celld
pinned to v0.6.2.
BREAKING CHANGE: ZoneStateStore is async and has delete();
SqliteZoneStateStore and distributed_game's `sqlite` feature are gone.
game_demos: ClusterOptions.celld is required and state_dir is removed
(`--state-dir` flag gone, `--celld-url`/CELLD_URL and
DISTRIBUTED_INTERNAL_SECRET required for grid, ripple and freeze);
World::standalone takes (host id, hibernate_after, guests).
[[tasks/gr-p37-celld-zone-state]]
The `crawler-roster/all` cell was a global singleton every new player
passed through (celld cannot enumerate cells). The crawler list is now
the `player_stats` read model projected from the Crawler cells' own
`crawler.mob_killed` domain events (cell outbox -> Queue -> relay ->
bus -> projector):
- `mobpit_crawler::stats` holds the read model, mutation and projection
shared by `mobpit-service` and the zone server.
- The zone server projects the bus into its own database
(`mobpit/directory.rs`, bus consumer group `mobpit-zone`,
`{data-dir}/read-models.db`); `/dev/mobpit/crawlers` and the board
rebuild list crawlers from it, the rebuild still reads kill histories
from each listed cell.
- The relay route and projection bus move to `common::relay` so other
demos can consume their cells' events.
- `MobPitWorld::start` opens the world with its projection;
`attach_projection_bus` is now `attach_projection`.
- The roster cell, `crawler.enlist`, `ROSTER_KIND`/`ROSTER_SHARD`/`ENLIST`
and `CrawlerCells::principals` are removed (celld 0.6.2 has no
`deleted_classes`, so the class leaves the v1 migration).
Refs [[tasks/gr-p32-mobpit-celld]]
Zone lease events had no consumer (the Queue retried them into the DLQ) and the grid map opened a `ZoneState` cell per zone, creating cells that were never saved (celld materialises a Durable Object's storage on any request, so a cell read cannot answer "not found" without it). - `ZoneLease` and `ZoneState` publish compact post-transition state: `ZoneLeaseStatus` on `zone.activated|released|lease_expired`, `ZoneSnapshotStatus` (no bytes) on `zone_state.saved|deleted`. - New `world::zone_status`: `zone_leases` / `zone_snapshots` read models, their projections, `projectors()` and `ZoneStatus::of` (active / hibernated / cold). Tested end to end on the reference host (outbox -> SQLite bus -> consumer service -> rows). - Cluster hosts serve the celld Queue relay, share a projection bus in `--data-dir` and each project it into their own zone status database. - `/dev/grid/map` reads the read model; only live tier/player counts are asked from the hosts. `spawn_cluster` is now async. Refs [[tasks/gr-p37-celld-zone-state]] [[tasks/gr-p36-mobpit-3d-services]]
Both hosts of a cluster demo open the shared `bus.db` at startup; switching a new file to WAL and creating the bus tables take the write lock, so one host could exit with `database is locked` (seen in the grid and ripple headless tests). `ProjectionBus::open` now retries that error for up to 10 s. The Mob Pit crash test keeps the server's stderr in its data dir and names it on failure; the presence test waits up to 20 s for a cold presence cell under load. Refs [[tasks/gr-p37-celld-zone-state]] [[tasks/gr-p32-mobpit-celld]]
Aggregates schedule work and read the host clock through the repository
pattern instead of cell-only APIs:
- `distributed::schedule`: `Schedule`, `ScheduleWrite`, `ArmedSchedule`,
`schedule_session`/`schedule_guard`, `Capability`/`Capabilities`
(were `cell_host::CellAlarm*`). `ctx.schedule`, `ctx.cancel_schedule`,
`ctx.scheduled_due` and `ctx.now()` replace `arm_alarm`,
`cancel_alarm`, `alarm_due` and `cell_now`.
- `CommitBatch.schedules` (was `alarms`), owned by the batch's one
stream, stored in the commit on every backend: cell `cell_alarms`,
in-memory per-stream table, SQL `aggregate_schedules` (migration 12;
relative due times by the database clock).
- Backends declare capabilities: cells dispatch schedules and stamp the
DO/store clock; in-memory (`with_clock`) and SQL (database clock)
store schedules but do not dispatch yet. Routes declare
`.requires(Capability::..)`; registration panics when the backend
lacks one, `ctx.schedule` refuses on undeclared routes.
- `AggregateCell::{armed,next,fire_due}_schedule*`; zone lease, alarm
worker, tests and docs ported.
- `distributed_game::command_sink::CommandFactSink<H: CommandHost>` with
`FactRoute`; `CelldFactSink` is that sink over a remote-only
`CelldCommandHost` (session-aware cell routes, forwarded `x-game-*`
stamp). Cells answer `CellDispatchResult::wait_path_wire`;
`CausalDispatchResult::replayed()`. A local-host test runs the celld
test's fact routes against an in-memory repository.
Refs [[tasks/gr-p38-portable-schedule-and-fact-sink]]
The e2e-ui celld profile defaulted CELLD_HTTP_PORT to 18080, which Zitadel's edge from `make up` already binds, so `make up && make up-celld-nats` failed. celld now defaults to 18880 in the e2e-ui, e2e-celld and tests/celld Makefiles. One celld version, in /.celld-version (0.6.2), checked by scripts/celld.sh (check prints the upgrade command; install is what CI runs). run.sh, the e2e-ui profile and every workflow (including the previously unpinned live job) use it. The binary is CELLD_BIN everywhere; the CELLD alias in alarms.mjs and the Rust support is gone. Also from A8: drop the dead .make-ui/.make-runner pid handling and the unused DATABASE_URL default, fix the "UI :5180" comment, pass --context to every kubectl call in dual-worktree-suite.sh, and replace the zitadel-ingestor sample query that used undeclared reverse joins. BREAKING CHANGE: the celld binary env var is CELLD_BIN (CELLD is no longer read); the profile's default celld port is 18880. [[tasks/gr-p40-examples-e2e-coverage]]
README pointed at a removed blob_move.rs handler (now blob-domain's move_dir.rs) and a tests/read_models/ dir that does not exist. application-composition.md and the e2e-ui README still named crates/service/src/service.rs, which became application.rs, host.rs and modules/. The gateway metrics doc now says where the load report is written versus the checked-in recorded run. [[tasks/gr-p40-examples-e2e-coverage]]
The handler samples used .causal(...) and .projected(), which do not exist; they now show the real portable_command! for todo.complete and the blob.move handler ending in .atomic(). The tag line listed Kafka, RabbitMQ, Keycloak and Authentik, which this app does not use. The optimism section said all three journeys hold the request before the wire; only chat and blob do, todo holds the response. [[tasks/gr-p40-examples-e2e-coverage]]
File labels named */src/commands.rs and service.rs, which became commands/*.rs and modules/graphql.rs. The host sample called build_graphql_engine and serve_with_oidc; the runner sample lacked three HostOptions fields; the blob tab told readers to `make wasm`, which does not exist (distributed build owns the pure). The celld tabs described a CelldTodoCommandHost that dual-writes locally and routes only create/complete; the real path is the framework CelldCommandHost with per-crate CelldRoutes for every Todo transition, and projections fed by the cell outbox through the Queue relay and NATS. Removes the unused allWalkthroughs/walkthroughById exports. [[tasks/gr-p40-examples-e2e-coverage]]
The cells worker and its Queue carry every demo's cells (Mob Pit,
The Freeze, zone leases and state), so mobpit-cells / mobpit-outbox
become game-demo-cells / game-demo-outbox (and -dlq). Docs no longer
say grid zones hibernate into SQLite, that the zone server drains the
outbox, or that only mobpit|leaderboard|sveltekit need celld. The
README Layout lists cells/, devcelld.rs, common/{celld,relay,
zone_status}.rs and helpers/celld.mjs; dev routes list
/dev/mobpit/presence; the demo count is eight. run.sh help prints the
whole header instead of stopping at line 39.
BREAKING CHANGE: the cells worker is game-demo-cells and its Queue
game-demo-outbox; existing local celld dev state under .data/ keyed by
the old worker name is not reused.
[[tasks/gr-p40-examples-e2e-coverage]]
`distributed contracts check|accept` was missing from ServiceCommands, so `hops service contracts` did not exist; a test now asserts the hosted command set equals the standalone one. Its --catalog default was contracts/catalog.json, a file that does not exist: make contracts-check failed and the PR gate silently skipped. Both now use distributed.contracts.json (the catalog `distributed build` reads), and the gate always runs. Root README: feature table adds metrics, otel, workers-rs, application-runtime/runtime and the gateway features; the Locking row no longer claims Redis; Project Structure lists every src module and the distributed_game*, distributed_cli, examples and e2e-celld trees. distributed_cli README documents contracts; js README lists ./replica/lazy; distributed_game's Cargo comment names celld_worker. BREAKING CHANGE: `distributed contracts check` defaults to --catalog distributed.contracts.json. [[tasks/gr-p40-examples-e2e-coverage]]
tests/e2e-ui/gitops/ (cluster providers, Auth/PSQL stacks, env apps) was superseded by .gitops/local and referenced nowhere. The contracts/deployment-plans/*.json fixtures and the dual-worktree suite (plus its HMR helper) were also unreferenced and never run. [[tasks/gr-p40-examples-e2e-coverage]]
The live celld job ran only todos and chat; blob, admin, anon and the unique-key spec never ran on celld. `make -C tests/e2e-celld test-browser` and CI now run every Playwright project against the celld-backed app. Locally all 21 tests pass on celld 0.6.2. Found while running it: - up-celld-nats needs esbuild for celld's bundler; it now falls back to js/node_modules/.bin/esbuild and says how to get one otherwise. - Zitadel's first boot took over three minutes here, past up.sh's 180 s wait; the wait is ZITADEL_WAIT_TRIES (default 180 x 2 s). - The app Postgres host port is E2E_APP_DB_PORT (default 5433). [[tasks/gr-p40-examples-e2e-coverage]]
Shared repositories stored ctx.schedule rows but nothing fired them, so scheduling routes could not register there. A ScheduleWorker now claims due rows in one statement (UPDATE ... IN (SELECT ... FOR UPDATE SKIP LOCKED) RETURNING on Postgres; SQLite serializes the write), dispatches each through a CommandHost with the trusted schedule session, and settles only the generation that fired. The command id is derived from (owner, name, generation): one ledger serves every aggregate there, so the owner is part of it. A worker that crashes after the command committed leaves its claim (claimed_until_ms, migration 0013) to lapse; the next worker re-dispatches the same id and the ledger replays it, so each arm fires once. `with_schedule_worker(config)` on SqliteRepository, PostgresRepository and InMemoryRepository returns the repository, which now declares Capability::Scheduling, plus the worker to spawn with the service's host. Poll interval, claim TTL and batch size are configurable (250 ms / 30 s / 32). Latency vs celld is documented. Tests (tests/schedule_worker): capability gate, fire-once and periodic re-arm, crash between commit and settle with a restarted process, two concurrent workers on one database, retry backoff; on SQLite and, with DATABASE_URL, Postgres (added to the Postgres CI job); plus the in-memory repository on a ManualClock. [[tasks/gr-p39-postgres-game-adapters]]
LeaderboardProjector's change listener only reached a GraphQL engine in the same process, so a projector elsewhere (Mob Pit's zone server owns its Redis boards) could not drive @LiVe in the GraphQL service. Board changes now travel on the application's bus, like every other cross-process fact: LeaderboardChangePublisher (feature leaderboard-background) is a projector listener that publishes a `distributed.leaderboard.changed` event ({ "board": name }) per changed board, coalescing bursts and retrying until the bus accepts it. GraphqlEngine::leaderboard_change_handlers(group) returns bus Handlers that republish `leaderboard:{board}` on the engine's ChangeHub; each GraphQL process subscribes under its own consumer group. A notice only names the board, so duplicates and replays push nothing new. Test: a projector and an engine on separate SQLite bus connections; the applied change pushes the new top-N to the subscriber. [[tasks/gr-p29-leaderboard-graphql]] [[tasks/gr-p40-examples-e2e-coverage]]
An application manifest whose surface had leaderboard roots failed to
build ("surface contract material"): SurfaceSpec carried no
leaderboard section, so the contract rebuilt from the spec never
matched the Surface IR's. SurfaceSpec now has `leaderboards` (the
canonical sections, absent when empty so existing manifests keep their
bytes and fingerprints), and the reconstruction includes them.
JS: the replica GraphQL transport gains `query(document, variables)`
and `live(document, variables, observer)` for documents that never
enter the replica (server-ranked boards), over the same endpoint,
credential and WebSocket. `@hops-ops/distributed/sveltekit` adds
`loadDistributedLeaderboard` (SSR load of a generated
`…Leaderboard` operation) and `useDistributedLeaderboard` /
`createDistributedLeaderboardStore` (a store seeded with the SSR result
that follows the operation's `@live` document).
BREAKING CHANGE: ReplicaGraphqlTransport requires `query` and `live`.
[[tasks/gr-p29-leaderboard-graphql]] [[tasks/gr-p40-examples-e2e-coverage]]
The SvelteKit leaderboard was a SQL query over player_stats. It is now the zone server's Redis XP board, served by mobpit-service as GraphQL board roots (xp/kills: Leaderboard, LeaderboardAround, MyRank) and rendered with the generated XpLeaderboardLeaderboard operation: SSR-loaded in the pit page load, then `@live`. The zone server's projector publishes `distributed.leaderboard.changed` on the projection bus (once attached); mobpit-service subscribes under its own consumer group and pushes. Display names still come from the player_stats read model (Leaderboard.graphql island). Board names and the Redis key prefix moved to mobpit_crawler so both processes agree; the zone server takes --redis-prefix / MOBPIT_REDIS_PREFIX. tests/projection.rs now also proves a board push that reaches the engine only through the bus notices; pit.test.mjs runs on Redis with a per-run prefix and checks the second player's board is live. BREAKING CHANGE: mobpit-service requires REDIS_URL, and `run.sh sveltekit` / `run.sh smoke` require REDIS_URL (CI sets it). [[tasks/gr-p40-examples-e2e-coverage]] [[tasks/gr-p29-leaderboard-graphql]]
The schedule claims migration (0013) made three version assertions stale: the SQLite and Postgres upgrade tests and the CLI's migration inventory order test. [[tasks/gr-p39-postgres-game-adapters]]
The crash test assumed the restarted worker's first pass ran within the crashed worker's 400 ms claim; opening and migrating a Postgres pool sometimes took longer. The claim is now 3 s and the test waits from the observed commit. The flaky-command counter was a process-wide static shared by the SQLite and Postgres variants running in parallel; each test now marks its own ticker id. [[tasks/gr-p39-postgres-game-adapters]]
With REDIS_URL exported for the whole smoke run, the other demos' tests (which assert their in-memory boards) got Redis too. smoke now requires REDIS_URL, unsets it, and passes it to the SvelteKit test only. [[tasks/gr-p40-examples-e2e-coverage]]
The target released its reservation in a spawned task after replying, so
a client that immediately returned to its source could still find the
transfer reserved ("transfer in progress"). Await the release first.
One swing can kill several mobs, so the hunt can stop at two settled kills while a third is still in flight; the board then jumps past the captured profile and never matches. Wait for acked inputs and settled kills first.
Exec-ing a just-written script can fail with ETXTBSY when another test thread forks while the script's write fd is open. Pass the script to /bin/sh so it is only read, never exec'd.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #280. Work for the game-services package (spec
specs/framework/game-services/v1/indexin the dungeon-crawler GitKB). Draft: more commits land here overnight.Done so far
FactCommandrejects amount/coins/gold/price/balance-style fields; list shared with ticket claims.FactStampgains emission time, activation epoch and config version;emission_orderorders facts across zones and handoffs, even with clock skew. Replay formatDGR3.#[derive(TicketExtension)], server-only claims (mintGameTicket({ extensions }),claims()callback); the browser body still cannot add claims.ZoneFactoryruns after the ownership claim with an optional allocation record; new protocol errorZONE_UNAVAILABLE.type Message+Zone::on_message, built-inKick,POST /admin/zone-message(admin secret, 404 inactive, 421 wrong host),ZoneInboxClient.ctx.arm_alarm/cancel_alarm, committed with the command; fired through the command ledger exactly once; shares the DO alarm with the outbox watchdog.SagaAction::FanOut(1..=64 commands); compensates only members that applied or timed out.scripts/check-wasm32-domains.sh+ required CI job, with a negative fixture.<board>Leaderboard,Around,MyRank(member from session),@liveupdates, generatedleaderboards.ts; real-Redis CI step.CelldFactSinkroutes facts by principal, field or lifecycle; queues while celld is down.ZoneLease+CellZoneOwnership; alarm publishes onezone.lease_expired;ctx.cell_now().FactStamp.seqorders facts stamped in the same tick; fixes the Mob Pit presence flake (16/20 failures before, 80/80 passes after). A failed socket write now notifies the zone.#[replicated(.., of = T)]+#[from]/#[from(path)],Project<V>,EntityId,Zone::entities(). Mob Pit has noreplicate(); a golden test shows identical Delta bytes over 900 ticks.#[command]on aggregate methods (gr-p17): inside#[sourced]impls; principal always from the verified session; golden test shows identical specs, SDL and manifest. Mob Pitcrawler.record_killmigrated.examples/game/cells(crawler, presence cells); kills reach the read model through cell outbox → celld Queue → relay → bus → projector → GraphQL@live.run.shstarts a privatecelld dev(CELLD_BIN, needs ≥ 0.6.2); CI installs celld v0.6.2.ZoneState:{zone}cell with fencing; snapshots over 64 KiB go to the worker's R2/RustFS bucket with a digest. Grid, Ripple, Freeze, Mob Pit, Leaderboard and SvelteKit run on celld; Arena and Gallery have no durable state.world::zone_status(active / hibernated / cold); the grid map reads it and never creates cells.ctx.schedule/cancel_schedule/scheduled_due/ctx.now()indistributed::schedule; schedule rows commit with the events on cells, in-memory and SQL (migration 12). Routes declare.requires(Capability::…), checked when the service is built.CommandFactSink<H: CommandHost>;CelldFactSink::celld(..)is that sink over celld./.celld-version(0.6.2) read byscripts/celld.shin every workflow; e2e celld defaults to :18880 so it runs beside Zitadel; README/walkthrough samples match the code; removed the orphaned gitops tree and dead scripts.distributed contracts checknow defaults todistributed.contracts.json(the PR gate was silently skipping); addedhops service contracts.make -C tests/e2e-celld test-browserand the CI live job run every project: 21 passed, 0 skipped.ScheduleWorkerclaims due rows withFOR UPDATE SKIP LOCKED(migration 13,claimed_until_ms) and dispatches throughCommandHost;with_schedule_worker(..)on SQLite/Postgres/in-memory repositories enablesCapability::Scheduling. 11 tests incl. crash-between-commit-and-settle exactly once and two concurrent workers, on SQLite and Postgres. Latency (about one poll interval) documented indocs/coordination-primitives.md.@liveleaderboards (gr-p29, gr-p40 C2):LeaderboardChangePublishersendsdistributed.leaderboard.changedon the bus; each GraphQL process republishes it. Manifests gain aleaderboardssection; JS addsloadDistributedLeaderboard/useDistributedLeaderboard. Mob Pit SvelteKit shows the Redis xp board live.Breaking (v5 unreleased): every
Zonedeclarestype Message;GameServer::newtakes aZoneFactory;GameServer::activateis async;Clock::unix_msrequired; sagacompensation/on_step_succeededtake a member index; in-processCellFactSinkis native-only;FactStamprequiresseq;Zone::replicatedefaults to empty andWorldView::addtakesimpl Into<EntityId>; removed thecell-sqlitefeature,AggregateCell::open_sqlite*andSqliteZoneStateStore;ZoneStateStoreis async; demo binaries require--celld-url;arm_alarm/cell_nowrenamed toschedule/now,CommitBatch.alarms→.schedules;spawn_clusteris async; celld binary only viaCELLD_BIN, e2e celld port 18880; cells worker renamedgame-demo-cells(queuegame-demo-outbox); contracts catalog default isdistributed.contracts.json;ReplicaGraphqlTransportrequiresqueryandlive; mobpit-service andrun.sh sveltekitneedREDIS_URL.Verified
cargo test --workspace --all-features --all-targets: 2604 pass. JS 490 pass. wasm32 domain check passes.examples/game/run.sh smokeOK (incl. SvelteKit Playwright). e2e-ui Playwright: 21 pass on Postgres and 21 pass on celld 0.6.2 + NATS.Known, not from this PR: js-client quality job (runtime.mjs assertion) and the celld storage job already failed at
cd89746b; occasional flakes in a JS command-runtime timing test, aDevCelldfree-port race and the admin force-archive spec.