Skip to content

perf(webapp): avoid unindexed fileId scan in get-background-worker-by-version#4245

Merged
d-cs merged 2 commits into
mainfrom
perf-background-worker-version-endpoint
Jul 13, 2026
Merged

perf(webapp): avoid unindexed fileId scan in get-background-worker-by-version#4245
d-cs merged 2 commits into
mainfrom
perf-background-worker-version-endpoint

Conversation

@d-cs

@d-cs d-cs commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

What

The GET /api/v1/projects/:projectRef/background-workers/:envSlug/:version endpoint loaded each file's tasks through the nested files.tasks relation. Prisma resolves that as a separate query:

SELECT id, slug, "fileId" FROM "BackgroundWorkerTask" WHERE "fileId" IN (...)

BackgroundWorkerTask.fileId is not indexed — the FK constraint exists, but Postgres does not auto-create an index for foreign keys — so on a large table this can only run as a sequential scan, which gets progressively slower as the table grows and was observed taking minutes per call in production.

The loader already loads every task for the worker via tasks: true, which uses the indexed workerId relation, and those rows already include fileId. This PR groups task slugs by fileId in memory from that already-loaded data and drops the files.tasks include entirely.

Behavior change (latent bug fix)

The response shape is unchanged, but there is a semantic correction for source files reused across worker versions (files are de-duplicated by @@unique([projectId, contentHash]), so one file row can be linked to many workers).

  • Before: file.tasks came from the BackgroundWorkerFile.tasks relation, i.e. every BackgroundWorkerTask with that fileId — across all workers sharing the file. So a worker's manifest could list tasks it doesn't actually have.
  • After: file.tasks is grouped from the queried worker's own tasks, so it reflects only that worker version's tasks.

Verified on a local DB: 460 files are referenced by tasks from more than one worker; of 6819 (worker, file) pairs, 6 differ — all one file where the old union leaked a task slug (cancellation-test) into worker versions that never had it. The new per-worker behavior is the correct one for a worker-version manifest. (Thanks to the automated review for flagging this.)

Analysis

Captured the exact SQL before/after by instrumenting Prisma against real data (a worker with 62 files):

  • Before: 5 statements, including the WHERE "fileId" IN (...) scan.
  • After: 4 statements; the fileId query is gone and the other four are identical.

EXPLAIN of the two access paths:

Before  WHERE "fileId" IN (...)
  Seq Scan on "BackgroundWorkerTask"
    Filter: ("fileId" = ANY (...))          -- reads the whole table, scales with table size

After   WHERE "workerId" IN (...)
  Index Scan using "BackgroundWorkerTask_workerId_slug_key"
    Index Cond: ("workerId" = ...)          -- bounded by matching rows, scale-independent

No new index is required: the workerId access path is already covered by the existing BackgroundWorkerTask_workerId_slug_key unique index.

Testing

  • pnpm run typecheck --filter webapp passes.
  • Query capture + EXPLAIN performed against a local database seeded with real worker/file/task data.

…-version

The endpoint loaded each file's tasks via the files.tasks relation, which
queries BackgroundWorkerTask by the unindexed fileId column and
sequential-scans a very large table for every request.

Group task slugs by fileId in memory from the tasks that are already loaded
via the indexed workerId relation, and drop the files.tasks include. The
response shape is unchanged.
@changeset-bot

changeset-bot Bot commented Jul 13, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 3a7905d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@coderabbitai

coderabbitai Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

The background worker version endpoint now loads files directly and builds a file ID-to-task-slug mapping from already-loaded background worker tasks. Each response file uses this mapping for its tasks list, while other response data remains unchanged. A change note documents the retrieval optimization and unchanged response content.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check ⚠️ Warning The PR description is detailed, but it omits required template sections like Closes #issue, checklist, changelog, and screenshots. Add the missing template sections: Closes #issue, completed checklist, testing steps, a short changelog, and screenshots or N/A notes.
✅ Passed checks (3 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the performance fix in the background-worker version endpoint.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch perf-background-worker-version-endpoint

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@d-cs d-cs self-assigned this Jul 13, 2026
@d-cs d-cs marked this pull request as ready for review July 13, 2026 15:02

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

Open in Devin Review

@pkg-pr-new

pkg-pr-new Bot commented Jul 13, 2026

Copy link
Copy Markdown

Open in StackBlitz

@trigger.dev/build

npm i https://pkg.pr.new/@trigger.dev/build@3a7905d

trigger.dev

npm i https://pkg.pr.new/trigger.dev@3a7905d

@trigger.dev/core

npm i https://pkg.pr.new/@trigger.dev/core@3a7905d

@trigger.dev/python

npm i https://pkg.pr.new/@trigger.dev/python@3a7905d

@trigger.dev/react-hooks

npm i https://pkg.pr.new/@trigger.dev/react-hooks@3a7905d

@trigger.dev/redis-worker

npm i https://pkg.pr.new/@trigger.dev/redis-worker@3a7905d

@trigger.dev/rsc

npm i https://pkg.pr.new/@trigger.dev/rsc@3a7905d

@trigger.dev/schema-to-json

npm i https://pkg.pr.new/@trigger.dev/schema-to-json@3a7905d

@trigger.dev/sdk

npm i https://pkg.pr.new/@trigger.dev/sdk@3a7905d

commit: 3a7905d

@d-cs d-cs merged commit e0b42a8 into main Jul 13, 2026
39 of 41 checks passed
@d-cs d-cs deleted the perf-background-worker-version-endpoint branch July 13, 2026 15:14
@github-actions github-actions Bot mentioned this pull request Jul 13, 2026
ericallam pushed a commit that referenced this pull request Jul 14, 2026
## Summary
2 new features, 11 improvements, 5 bug fixes.

## Breaking changes
- Trigger.dev v3 is no longer supported. For self-hosted deployments,
4.5.0 is the last version we officially support for running v3; stay on
4.5.0 or upgrade to v4. v3 triggers, batch triggers, reschedules, and
deploys now return a clear upgrade message instead of running.
([#4236](#4236))

## Improvements
- You can now mark environment variables synced via the `syncEnvVars`
build extension as secrets. Return `{ name, value, isSecret: true }`
from your callback and those variables are stored redacted in the
dashboard, just like manually created secret env vars.
([#4203](#4203))
- Remove the legacy `--mcp` and `--mcp-port` options from the `dev`
command. Run the dedicated `trigger mcp` command to start the
Trigger.dev MCP server.
([#4246](#4246))
- Removed the unused `ResourceMonitor` export from
`@trigger.dev/core/v3/serverOnly`. It was a server-side logging helper
with no remaining consumers.
([#4244](#4244))
- Removed the unused `@trigger.dev/core/v3/zodNamespace` export and the
legacy v3 socket message schemas. These were only used by the
now-retired v3 engine and have no v4 consumers.
([#4236](#4236))

## Bug fixes
- Fix a `chat.agent` message-loss race where sending a message right
after an action (such as an undo) could drop the follow-up's response
from the UI until a refresh.
([#4234](#4234))

## Server changes

These changes affect the self-hosted Docker image and Trigger.dev Cloud:

- Added `EVENT_REPOSITORY_POSTGRES_WRITES_DISABLED` to skip all
PostgreSQL task-event writes for deployments that store task events in
ClickHouse. Leave it off unless `EVENT_REPOSITORY_DEFAULT_STORE` is
`clickhouse_v2`, otherwise task events are lost.
([#4242](#4242))
- Promo credits: a /promo signup landing page, redeeming a promo code
when a new org selects a plan, and showing remaining credits on the
usage page.
([#4138](#4138))
- Speed up retrieving a background worker by version. The endpoint no
longer runs a slow lookup that scanned the full task table for large
deployments; it now reuses data it already loads, so the response is the
same but returns much faster.
([#4245](#4245))
- Clearer login error when an email address is blocked by the
WHITELISTED_EMAILS setting: the message now explains the address isn't
allowed on this instance instead of the ambiguous "This email is
unauthorized".
([#4220](#4220))
- Make the native build server the default in project build settings.
It's now opt-out, stored as a new `disableNativeBuildServer` key. Also
clarifies in the UI that build settings apply to GitHub-triggered and
native build server deployments.
([#3980](#3980))
- Optionally process high-volume telemetry ingestion in parallel for
higher throughput under heavy load by setting
`OTEL_TRANSFORM_WORKER_POOL_ENABLED=1`. Off by default.
([#4232](#4232))
- Add a `REALTIME_BACKEND_DEFAULT` env var to choose the default
realtime backend (`electric`, `native`, or `shadow`) for environments
whose org has no per-org override. Defaults to `electric`, so existing
behavior is unchanged.
([#4231](#4231))
- Clarified on the Regions page that a region only affects where your
runs execute, not where your data is stored. This shows as a tooltip on
the Location column and in the confirmation dialog when you change your
default region.
([#4226](#4226))
- Improved the reliability of how run data is read and written.
([#4237](#4237))
- Fixed stale login errors: an error from a previous login attempt (for
example a rejected email address) no longer keeps reappearing on the
login page and no longer makes later, successful attempts look like they
failed.
([#4220](#4220))
- The Errors page now shows better details for each error. Errors that
don't carry a message — such as errors thrown without a message, or
values thrown that aren't `Error` objects — get a meaningful title
instead of all reading "Unknown error", and are grouped by their name
(or value) rather than collapsed into a single group. The error type now
shows the actual error name, and stack traces now appear where
previously they were missing.
([#4225](#4225))
- Return a clear client error when SSO form submissions use an
unsupported content type
([#4238](#4238))
- Query page: extracting fields from a run's output with JSON functions
(such as JSONExtractString or JSONExtractInt) no longer fails with an
"illegal type: JSON" error.
([#4221](#4221))

<details>
<summary>Raw changeset output</summary>

# Releases
## @trigger.dev/build@4.5.4

### Patch Changes

- You can now mark environment variables synced via the `syncEnvVars`
build extension as secrets. Return `{ name, value, isSecret: true }`
from your callback and those variables are stored redacted in the
dashboard, just like manually created secret env vars.
([#4203](#4203))
- Updated dependencies:
  - `@trigger.dev/core@4.5.4`
## trigger.dev@4.5.4

### Patch Changes

- Remove the legacy `--mcp` and `--mcp-port` options from the `dev`
command. Run the dedicated `trigger mcp` command to start the
Trigger.dev MCP server.
([#4246](#4246))
- Updated dependencies:
  - `@trigger.dev/core@4.5.4`
  - `@trigger.dev/build@4.5.4`
  - `@trigger.dev/schema-to-json@4.5.4`
## @trigger.dev/core@4.5.4

### Patch Changes

- Removed the unused `ResourceMonitor` export from
`@trigger.dev/core/v3/serverOnly`. It was a server-side logging helper
with no remaining consumers.
([#4244](#4244))
- Removed the unused `@trigger.dev/core/v3/zodNamespace` export and the
legacy v3 socket message schemas. These were only used by the
now-retired v3 engine and have no v4 consumers.
([#4236](#4236))
## @trigger.dev/python@4.5.4

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/sdk@4.5.4`
  - `@trigger.dev/core@4.5.4`
  - `@trigger.dev/build@4.5.4`
## @trigger.dev/react-hooks@4.5.4

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.4`
## @trigger.dev/redis-worker@4.5.4

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.4`
## @trigger.dev/rsc@4.5.4

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.4`
## @trigger.dev/schema-to-json@4.5.4

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.4`
## @trigger.dev/sdk@4.5.4

### Patch Changes

- Fix a `chat.agent` message-loss race where sending a message right
after an action (such as an undo) could drop the follow-up's response
from the UI until a refresh.
([#4234](#4234))
- Updated dependencies:
  - `@trigger.dev/core@4.5.4`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants