feat: DSC support for mongodb runner by mpobrien · Pull Request #822 · mongodb-js/devtools-shared

mpobrien · 2026-07-15T19:33:34Z

Adds the ability to launch DSC clusters backed by SLS.

New disaggregatedStorage cluster option (all topologies): starts a docker compose project for the storage backend, waits for readiness, runs per-shard setup, and injects disaggregatedStorageConfig/disaggregatedStorageEnabled into every mongod. Compose project is shared across shards and torn down with the cluster (including on failed startup).
SLS helpers: createSLSDisaggregatedStorageOptions() handles the full flow for the SLS multi-cell compose file from a server checkout — env/port allocation (services parsed from the compose file itself), readiness polling, per-shard StartLog, encryption key file, and auto-generated disaggregatedStorageConfig (incl. replSetConfig from pre-allocated member ports).
DSC-specific behavior: replset defaults to 2 nodes, replSetInitiate is skipped (config comes from startup parameters), and local-db metadata tracking is disabled (not writable on DSC).
New CLI flags: --slsCompose/--slsImageTag (full SLS flow), --disaggregatedStorageCompose/--disaggregatedStorageConfig (custom backends), and --downloadUrl for using a direct mongod tarball URL
(also added to mongodb-downloader, cached by URL).
Misc: docker compose output + download progress in debug logs, mongodb-runner start prints allocated SLS service ports.

Sample invocation:

node bin/runner.js start -t replset \
  --slsCompose=/home/ubuntu/mongo/buildscripts/modules/atlas/sls-multicell-docker-compose.yml \
  --slsImageTag=dfb12c1092695ae79f6ff3c5b6c31ff7417e4843 \
  --binDir=/mnt/mac/Users/mikeo/ubuntu-mongod/dist-test/bin/ \
  --debug --logDir=/tmp/mongodb-runner-logs

Open Questions

Checklist

I have signed the Contributor License Agreement (https://www.mongodb.com/legal/contributor-agreement)
New tests and/or benchmarks are included
Documentation is changed or added

Copilot

Pull request overview

Adds disaggregated storage (SLS) support to mongodb-runner, including orchestration of a docker compose-backed storage layer, pre-allocated ports for mongod members, and new CLI/config surface to drive it. It also extends mongodb-downloader to support downloading MongoDB binaries from a direct tarball URL (with progress logging).

Changes:

Add disaggregated storage abstractions (DisaggregatedStorageOptions, SLS helpers) and wire them into cluster startup/teardown.
Add docker compose lifecycle helper and CLI flags for SLS/disaggregated storage setup.
Add downloadUrl support to the MongoDB downloader and runner, enabling custom server builds.

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
packages/saslprep/src/code-points-data.ts	Updates embedded saslprep code point data blob.
packages/mongodb-runner/src/util.ts	Adds `allocatePort()` helper for pre-allocating member ports.
packages/mongodb-runner/src/util.js	Adds JS runtime utility helpers (parallel helpers, safePromiseAll, connection string helpers).
packages/mongodb-runner/src/tls-helpers.js	Adds TLS client key material helper for runner-managed TLS setups.
packages/mongodb-runner/src/sls.ts	Introduces SLS compose parsing, env/port allocation, readiness, and config generation helpers.
packages/mongodb-runner/src/runner-helpers.js	Adds JS runtime helpers for CLI commands (start/stop/prune/exec).
packages/mongodb-runner/src/oidc.js	Adds JS runtime OIDC mock provider subprocess management.
packages/mongodb-runner/src/mongoserver.ts	Adds disagg detection and skips local-db metadata tracking for disagg servers.
packages/mongodb-runner/src/mongoserver.js	Adds JS runtime implementation of MongoServer (process management, metadata, logging).
packages/mongodb-runner/src/mongologreader.js	Adds JS runtime log parsing/stream filtering for port/build info extraction.
packages/mongodb-runner/src/mongocluster.ts	Wires in disaggregated storage (compose lifecycle, per-shard setup, port preallocation) and `downloadUrl`.
packages/mongodb-runner/src/mongocluster.spec.js	Adds JS test coverage for cluster behaviors and CLI plumbing.
packages/mongodb-runner/src/mongocluster.js	Adds JS runtime implementation of MongoCluster including docker compose integration.
packages/mongodb-runner/src/index.ts	Exports new disagg/SLS and docker compose APIs from the package entrypoint.
packages/mongodb-runner/src/index.js	Adds JS runtime entrypoint exports for main APIs and CLI helpers.
packages/mongodb-runner/src/docker-compose.ts	Adds typed docker compose project wrapper (env + project naming + down --volumes).
packages/mongodb-runner/src/docker-compose.js	Adds JS runtime docker compose wrapper.
packages/mongodb-runner/src/cli.ts	Adds CLI flags for `downloadUrl`, SLS compose/image tag, and generic disagg compose/config.
packages/mongodb-runner/src/cli.spec.js	Adds JS tests for CLI workflows (start/stop/ls/prune/exec/config).
packages/mongodb-runner/src/cli.js	Adds JS runtime CLI implementation via yargs.
packages/mongodb-runner/README.md	Adds documentation pointer for disaggregated storage clusters.
packages/mongodb-runner/examples/sls-replset.js	Adds example for starting an SLS-backed replica set with a custom mongod build.
packages/mongodb-downloader/src/index.ts	Adds `downloadUrl` support, better error handling, and download progress logging.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

mpobrien · 2026-07-16T13:16:06Z

+import { spawn } from 'child_process';
+import { once } from 'events';
+import { parseCLIArgs, OIDCMockProvider } from '@mongodb-js/oidc-mock-provider';
+import { debug } from './util';
+if (process.env.RUN_OIDC_MOCK_PROVIDER !== undefined) {
+  (async function main() {
+    const uuid = crypto.randomUUID();
+    debug('starting OIDC mock provider with UUID', uuid);


Fixed in ecef1cf — the compiled .js files in src/ were committed accidentally and have been removed; the .ts source now imports randomUUID explicitly from 'crypto'.

mpobrien · 2026-07-16T13:16:07Z

+        return; // process already exited
+    }
+    if (!this.issuer || !this.uuid) return;
+    await fetch(new URL(this.issuer, `/shutdown/${this.uuid}`));


Fixed in ecef1cf — arguments swapped to new URL(/shutdown/${uuid}, this.issuer) in oidc.ts (the stale compiled .js files were removed).

mpobrien · 2026-07-16T13:16:09Z

+    throw new AggregateError(
+      [rejected.map((r) => r.reason)],
+      `${rejected.length} errors: ${rejected.map((r) => r.reason).join(', ')}`,
+    );


Fixed in ecef1cf — the errors array is no longer nested; AggregateError.errors now contains the individual errors.

mpobrien · 2026-07-16T13:16:10Z

+      if (disaggregatedStorage) {
+        // Pre-allocate member ports so the storage configuration (which
+        // embeds the replica set config, including member host:ports) can be
+        // computed before the servers start.
+        const members: ShardMemberDescriptor[] = [];


Fixed in ecef1cf — ShardMemberDescriptor now carries arbiterOnly, populated from the replica set member options, and the generated DSC replSetConfig includes it for each member.

mpobrien · 2026-07-16T13:16:12Z

+  isClosed() {
+    // Return true if and only if there are no running sub-clusters/servers
+    // eslint-disable-next-line @typescript-eslint/no-unused-vars
+    for (const _ of this.children()) return true;
+    return true;
+  }


Fixed in ecef1cf — isClosed() now returns false while child servers/shards exist (the stale compiled .js copy was removed).

- flatten AggregateError errors array in safePromiseAll - fix isClosed() to return false while child servers/shards are running - fix reversed new URL() arguments in OIDC mock provider shutdown - import randomUUID explicitly in oidc.ts - include arbiterOnly in ShardMemberDescriptor and generated DSC replSetConfig - add docs/disaggregated-storage.md

gribnoysup

The high level direction this is going looks good to me. I didn't do a thorough review yet as this is a draft, but a couple of things already caught my eye, so I left a couple of comments here and there. Happy to discuss those more

gribnoysup · 2026-07-23T09:33:55Z

+  /**
+   * A direct URL to a MongoDB tarball to download. If set, `version` and
+   * `downloadOptions` are ignored and no download URL lookup is performed.
+   */
+  downloadUrl?: string;


I think we should think about some different approach for adding this functionality in the downloader: the issue I see here is that while downloadOptions will be ignored, we will still consider them somewhat when returning the DownloadResult metadata, which from the consumer perspective might put you in a weird state where what you download doesn't match the returned options at all

gribnoysup · 2026-07-23T09:41:05Z

+              try {
+                return JSON.parse(argv.disaggregatedStorageConfig ?? '');
+              } catch {
+                return argv.disaggregatedStorageConfig ?? '';


Should this continue if we couldn't parse the arg?

gribnoysup · 2026-07-23T10:16:58Z

+    // Internal parameter used when starting the shards of a sharded cluster:
+    // the parent cluster owns the docker compose project, and the shards
+    // inherit the DSC context from it.
+    _internal?: {
+      disaggregatedStorage: DisaggregatedStorageOptions;
+      shardContext: { index: number; isConfigServer: boolean };
+    },


I find it a bit clunky that we're expressing this logic via an "internal" parameter on a public interface. If the idea here is that we want to know that the MongoCluster is a child of another MongoCluster that might be a disagg one, maybe we can model it that way and just pass a parent cluster as a second param here (and expose whatever we need from the parent so that the child cluster can read it)

Moved the implementation for this into private static _start(options, internal). public start(options) takes one argument again, and shard recursion goes through the private method

gribnoysup · 2026-07-23T10:23:38Z

+        cluster,
+        options,
+        disaggregatedStorage,
+        _internal?.shardContext ?? { index: 0, isConfigServer: false },


This fallback value is kinda weird: it looks like it's generic enough to apply to all shards we create, but it's gated by disagg check, so if someone reads it while creating a non-disagg cluster it will actually report state that is contradictory (a config server we're trying to start will read isConfig: false for example). Can we either rename this and make it optional to make sure it's clear it's disagg-only, or just make it generic enough so that the value is correct for all cases?

Changed this so that shard context is now always passed down for shards (not just in DSC mode), so the { index: 0, isConfigServer: false } default only applies to top-level clusters, where it's accurate.

gribnoysup · 2026-07-23T10:26:52Z

+    .option('slsCompose', {
+      type: 'string',
+      describe:
+        'Path to an SLS multi-cell docker-compose.yml; launches the SLS DSC project and configures mongod to use it (requires a DSC-capable mongod via --binDir or --downloadUrl)',
+    })
+    .option('slsImageTag', {
+      type: 'string',
+      describe:
+        'SLS docker image tag to use with --slsCompose (e.g. the pinned_sls_commit from the server repo manifest)',
+    })
+    .option('disaggregatedStorageCompose', {
+      type: 'string',
+      describe: 'Path to docker-compose.yml for the DSC backend',
+    })
+    .option('disaggregatedStorageConfig', {
+      type: 'string',
+      describe:
+        'JSON value for the disaggregatedStorageConfig setParameter on each mongod',
+    })


If you should either provide sls or disaggregated args (based on the logic I see below), maybe worth validating this case aborting if all these are provided?

Ended up stripping these out since there's no real reason to expose disaggregatedStorageConfig, the correct value for each mongod gets generated internally and there's likely no real use case to need to override it.
disaggregatedStorageCompose is redundant with slsCompose, so I just dropped it. With the remaining set of args, its safe/valid for them to all coexist, so no additional validation should be needed.

mpobrien · 2026-07-23T17:50:39Z

Thanks for the in-depth review @gribnoysup ! Will work through these, I might have a few follow-up questions.

autarch · 2026-08-06T16:57:58Z

Is there anything I can help with here? I need this to add DSC tests to the DB Tools.

mpobrien · 2026-08-11T02:27:32Z

@gribnoysup sorry for the delay, I addressed the comments.

mpobrien · 2026-08-11T19:20:43Z

@autarch until this gets merged you can check out the branch and use that to get started on using it for tools testing?

gribnoysup

A couple of small things, but otherwise looks good to me. I couldn't test the new cluster type myself locally, but at the very least the package tests are passing and using the new version in devtools repos also still runs all the tests correctly, so that's a good sign

gribnoysup · 2026-08-12T13:31:16Z

+  // clusters, for which it is accurate.
+  private static async _start(
+    { ...options }: MongoClusterOptions,
+    internal:


Can we give this a better name? Internal doesn't tell much when reading through the code. I guess this is like parentClusterContext or something?

gribnoysup · 2026-08-12T13:45:10Z

    useLockfile,
-  }: MongoDBDownloaderOptions): Promise<DownloadResult> {
+    downloadUrl,
+  }: MongoDBDownloaderOptions): Promise<DownloadResult | DownloadUrlResult> {


We should add the same function overloads as you added below, otherwise it changes the interface here in a way that would require every user to always manually assert whether or not the result contains expected values

gribnoysup · 2026-08-12T13:46:26Z

+      );
+    }
+    const totalBytes = +(response.headers.get('content-length') ?? '');
+    const totalMB = totalBytes ? (totalBytes / 1048576).toFixed(1) : null;


Nit: can we move the number to a var so that it's very clear immediately what it is and we always consistently use the same number below?

autarch · 2026-08-12T15:38:13Z

@autarch until this gets merged you can check out the branch and use that to get started on using it for tools testing?

Yes, that's what I'm doing.

mpobrien added 3 commits July 15, 2026 13:35

DSC support for mongodb-runner

1484845

wip

60e8af4

wip

5908876

Copilot AI review requested due to automatic review settings July 15, 2026 19:33

Copilot started reviewing on behalf of mpobrien July 15, 2026 19:34 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

wip

89be729

mpobrien changed the title ~~DSC support for mongodb runner~~ feat: DSC support for mongodb runner Jul 15, 2026

gribnoysup reviewed Jul 23, 2026

View reviewed changes

use partial type for direct download URL returns

caf85a1

mpobrien requested a review from gribnoysup August 11, 2026 02:27

mpobrien marked this pull request as ready for review August 11, 2026 02:27

gribnoysup requested changes Aug 12, 2026

View reviewed changes

Uh oh!

Conversation

mpobrien commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Open Questions

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gribnoysup left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mpobrien Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mpobrien commented Jul 23, 2026

Uh oh!

autarch commented Aug 6, 2026

Uh oh!

mpobrien commented Aug 11, 2026

Uh oh!

mpobrien commented Aug 11, 2026

Uh oh!

gribnoysup left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

autarch commented Aug 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mpobrien commented Jul 15, 2026 •

edited

Loading

mpobrien Aug 11, 2026 •

edited

Loading