MIGRATION-860 Hot Doc Check script

[sababich]

PR Title
Add hot document spread check script and documentation updates

Summary
This PR adds a script to identify hot documents that may explain high applier spread disparity during migrations. It also documents it in the toolbox index.

Problem / Context
When diagnosing slow CEA phase, it is useful to quickly detect whether one or a few documents are disproportionately receiving writes in a recent time window. This script combines spread and throughput gates.

What Changed

Added hot document analyzer script:
migration/toolbox/hotDocSpreadCheck/hot-doc-spread-check.js

Added detailed tool documentation:
migration/toolbox/hotDocSpreadCheck/README.md

Added toolbox index entry for this new tool:
migration/toolbox/README.md

Key script behavior

• Watches change stream events for insert/update/replace/delete.
• Aggregates per-document counts and computes:
  • event share
  • spread disparity estimate
  • per-document CPS
  • total-window CPS
• Returns hot document candidates only when all configured gates pass.

[Copilot]

Pull request overview

Adds a new mongosh-based diagnostic tool to detect “hot” documents by sampling recent change-stream writes and applying spread-disparity + throughput gates, and wires it into the migration toolbox documentation.

Changes:

• Added hot-doc-spread-check.js script to scan change streams and compute per-document hotness metrics with configurable thresholds and output formats.
• Added detailed tool README documenting usage, tuning, and output interpretation.
• Updated the toolbox index to include the new tool entry.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
migration/toolbox/README.md	Adds an index entry for the new Hot Doc Spread Check tool.
migration/toolbox/hotDocSpreadCheck/README.md	Documents purpose, usage patterns, tuning guidance, and sample output for the tool.
migration/toolbox/hotDocSpreadCheck/hot-doc-spread-check.js	Implements the change-stream sampling, aggregation, gating logic, and JSON/Markdown reporting.

Comments suppressed due to low confidence (1)

migration/toolbox/hotDocSpreadCheck/hot-doc-spread-check.js:409

• Same issue as JSON output: prefixing with "./" prevents writing to absolute paths and is unnecessary for relative paths.

    fs.writeFileSync("./" + markdownPath, buildMarkdownReport());

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[niccottrell]

@sababich is this ./ intentional/required? Otherwise I tend to access with Copilot here.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

[niccottrell]

Agree that we should be able to force processing of internal namespaces somehow.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (2)

migration/toolbox/hotDocSpreadCheck/hot-doc-spread-check.js:403

• outputFile is documented as an output path, but the script always prefixes it with "./" when writing. This breaks absolute paths (e.g., "/tmp/out.json" becomes ".//tmp/out.json") and is inconsistent with other toolbox scripts that write to the provided path directly (e.g., getBusiestCollection/get-busiest-collections.js:187, probIndexesComplete/probIndexesComplete.js:98).

    fs.writeFileSync(
      "./" + jsonPath,
      EJSON.stringify(output, null, 2, { relaxed: false })
    );

migration/toolbox/hotDocSpreadCheck/hot-doc-spread-check.js:411

• Markdown output is also written with a forced "./" prefix, which prevents using absolute output paths and diverges from other toolbox scripts that write to the given path directly.

  if (outputFormat === "markdown" || outputFormat === "both") {
    const markdownPath = outputFormat === "markdown" ? deriveOutputPath("markdown") : "hot-doc-spread-check.md";
    fs.writeFileSync("./" + markdownPath, buildMarkdownReport());
    print(`Wrote Markdown results to ${markdownPath}`);
  }

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

migration/toolbox/hotDocSpreadCheck/hot-doc-spread-check.js:186

• The "caught-up" stop condition is checked after fetching an event but the event is still counted/aggregated before the loop exits. This can include at least one post-open (out-of-window) change-stream event in the sample, skewing counts and potentially tripping gates unexpectedly.

    matchedEventsSeen++;
    lastEventAt = Date.now();
    lastClusterTime = evt.clusterTime;

    if (evt.clusterTime && evt.clusterTime.t * 1000 >= stopAtMs) {
      stopReason = "caught-up";
    }

migration/toolbox/hotDocSpreadCheck/README.md:5

• PR description says the detailed documentation was added as hot_doc_spread_check_readme.md, but the PR actually introduces hotDocSpreadCheck/README.md and the toolbox index links to that. Consider updating the PR description (or renaming files/links) to avoid confusion for readers.

# Hot Doc Spread Check

## Purpose

`hot-doc-spread-check.js` is a mongosh script that scans recent change activity and identifies documents that are hot enough to plausibly explain high applier skew.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (3)

migration/toolbox/hotDocSpreadCheck/hot-doc-spread-check.js:401

• Writing output files with a hard-coded "./" prefix breaks absolute output paths (e.g. "/tmp/report.json" becomes ".///tmp/report.json"). It also makes it harder to intentionally write outside the current working directory.

      "./" + jsonPath,

migration/toolbox/hotDocSpreadCheck/hot-doc-spread-check.js:409

• Same output-path issue for Markdown: prefixing with "./" prevents using absolute paths for markdownPath.

    fs.writeFileSync("./" + markdownPath, buildMarkdownReport());

migration/toolbox/hotDocSpreadCheck/README.md:5

• PR description says the tool documentation was added as hot_doc_spread_check_readme.md, but the PR adds migration/toolbox/hotDocSpreadCheck/README.md instead. Please update the PR description (or rename the file) so it matches the actual change.

# Hot Doc Spread Check

## Purpose

`hot-doc-spread-check.js` is a mongosh script that scans recent change activity and identifies documents that are hot enough to plausibly explain high applier skew.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

migration/toolbox/hotDocSpreadCheck/hot-doc-spread-check.js:410

• The script prepends "./" to the configured output file paths. This breaks absolute paths (e.g. "/tmp/out.json" becomes ".//tmp/out.json") and is inconsistent with other toolbox scripts that pass the filename directly (e.g. migration/toolbox/getBusiestCollection/get-busiest-collections.js:187). Write to the path as provided instead of forcing a relative prefix.

  if (outputFormat === "json" || outputFormat === "both") {
    const jsonPath = outputFormat === "json" ? deriveOutputPath("json") : DEFAULTS.outputFile;
    fs.writeFileSync(
      "./" + jsonPath,
      EJSON.stringify(output, null, 2, { relaxed: false })
    );
    print(`Wrote JSON results to ${jsonPath}`);
  }

  if (outputFormat === "markdown" || outputFormat === "both") {
    const markdownPath = outputFormat === "markdown" ? deriveOutputPath("markdown") : "hot-doc-spread-check.md";
    fs.writeFileSync("./" + markdownPath, buildMarkdownReport());
    print(`Wrote Markdown results to ${markdownPath}`);

[niccottrell]

@sababich Copilot has raised some good points that merit your consideration before we approve this.

[niccottrell]

@sababich is this ./ intentional/required? Otherwise I tend to access with Copilot here.

[niccottrell]

Agree that we should be able to force processing of internal namespaces somehow.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (3)

migration/toolbox/hotDocSpreadCheck/hot-doc-spread-check.js:204

• The event-processing loop always skips admin, config, local, and system.* namespaces, even when the user explicitly sets MONITOR_ARGS.namespaces to target one of them. This makes it impossible to analyze internal namespaces despite the $match stage allowing them when namespaces is non-empty.

    if (evt.ns.db === "admin" || evt.ns.db === "config" || evt.ns.db === "local" || evt.ns.coll.startsWith("system.")) {
      if (stopReason === "caught-up") break;
      continue;
    }

migration/toolbox/hotDocSpreadCheck/hot-doc-spread-check.js:180

• idleMs is validated with allowZero=true, but the idle-timeout check treats idleMs: 0 as “timeout immediately after the first matched event”, not “disable idle timeout”. Either disallow 0 in validation or gate the check on CFG.idleMs > 0 so 0 behaves as a real ‘no idle timeout’ setting.

This issue also appears on line 201 of the same file.

    if (Date.now() - lastEventAt > CFG.idleMs && matchedEventsSeen > 0) {
      stopReason = "idle-timeout";
      break;

migration/toolbox/hotDocSpreadCheck/README.md:260

• This README contradicts itself about internal namespaces: it says internal DBs/system collections are excluded only when namespaces is empty, but later states internal namespaces are always ignored. Please make this consistent with the intended behavior (especially if you want to allow explicitly-requested internal namespaces).

Events without a document `_id` are ignored.

Internal namespaces are always ignored: `admin`, `config`, `local`, and `system.*`.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (1)

migration/toolbox/hotDocSpreadCheck/hot-doc-spread-check.js:81

• namespaces accepts any db.coll, but internal namespaces (admin, config, local, system.*) are always ignored later in the event loop. If a user explicitly passes an internal namespace, the run will produce empty/low results without explaining why. Consider failing fast during namespace parsing with a clear error message (or, alternatively, introduce an explicit opt-in flag to include internal namespaces).

    const dot = ns.indexOf(".");
    if (dot <= 0 || dot === ns.length - 1) {
      throw new Error(`Invalid namespace "${ns}". Expected format: database.collection`);
    }
    return { db: ns.slice(0, dot), coll: ns.slice(dot + 1) };

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (2)

migration/toolbox/hotDocSpreadCheck/hot-doc-spread-check.js:55

• appliers represents a count, but the current validation only enforces a numeric value >= 2. Allowing non-integers can produce confusing threshold math (e.g., sqrt(appliers-1)) and makes the configuration semantics unclear. Consider enforcing that appliers is an integer >= 2.

  if (CFG.appliers < 2) {
    throw new Error(`Invalid appliers: ${CFG.appliers}. Expected a value of at least 2.`);
  }

migration/toolbox/hotDocSpreadCheck/README.md:366

• The README describes caught-up as having replayed forward to the “now” at stream open, but the script only sets caught-up after it observes an event with clusterTime >= stopAtMs. On quiet workloads (or windows with no matching events), this condition may never occur and the script can stop with run-ms-exceeded even though it is already caught up. Consider clarifying this behavior so users interpret stop reasons correctly.

* `caught-up`
  * it has replayed forward to the “now” that existed when the stream was opened
* `idle-timeout`
  * no matching events arrive for `idleMs` after activity has started
* `run-ms-exceeded`
  * hard max runtime reached

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (3)

migration/toolbox/hotDocSpreadCheck/hot-doc-spread-check.js:209

• Even if internal namespaces were allowed through config validation, the event loop unconditionally skips events from admin, config, local, and system.*, so internal namespaces can never be analyzed. If the intent is to optionally support internal namespaces when explicitly requested, gate this filter behind an opt-in config flag (keeping the current default behavior).

    if (!evt.ns || !evt.ns.db || !evt.ns.coll) {
      if (stopReason === "caught-up") break;
      continue;
    }

    if (evt.ns.db === "admin" || evt.ns.db === "config" || evt.ns.db === "local" || evt.ns.coll.startsWith("system.")) {
      if (stopReason === "caught-up") break;
      continue;
    }

migration/toolbox/hotDocSpreadCheck/README.md:5

• PR description says the documentation file added is migration/toolbox/hotDocSpreadCheck/hot_doc_spread_check_readme.md, but the repo change adds migration/toolbox/hotDocSpreadCheck/README.md (and the toolbox index links to hotDocSpreadCheck/README.md). Update the PR description (or rename the file) to avoid confusion for users trying to locate the docs.

# Hot Doc Spread Check

## Purpose

`hot-doc-spread-check.js` is a mongosh script that scans recent change activity and identifies documents that are hot enough to plausibly explain high applier skew.

migration/toolbox/hotDocSpreadCheck/hot-doc-spread-check.js:86

• parseNamespace() rejects admin, config, local, and system.*, which prevents analyzing internal namespaces even when a user explicitly passes them via MONITOR_ARGS.namespaces. Earlier review feedback indicated we should be able to force processing internal namespaces; consider adding an explicit opt-in (e.g., allowInternalNamespaces: true) that bypasses this validation (and keeps the current behavior as the default).

  function parseNamespace(ns) {
    const dot = ns.indexOf(".");
    if (dot <= 0 || dot === ns.length - 1) {
      throw new Error(`Invalid namespace "${ns}". Expected format: database.collection`);
    }
    const dbName = ns.slice(0, dot);
    const collName = ns.slice(dot + 1);
    if (dbName === "admin" || dbName === "config" || dbName === "local" || collName.startsWith("system.")) {
      throw new Error(`Invalid namespace "${ns}". Internal namespaces are not supported: admin, config, local, and system.*`);
    }
    return { db: dbName, coll: collName };

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (1)

migration/toolbox/hotDocSpreadCheck/README.md:104

• The example output doesn’t match the script’s actual console message. The script prints Wrote JSON results to ... (hot-doc-spread-check.js:429), but the README example says Wrote results to ..., which can be confusing when comparing expected vs actual output.

Wrote results to hot-doc-spread-check.json

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (4)

migration/toolbox/hotDocSpreadCheck/README.md:373

• The script writes the report using EJSON.stringify(..., { relaxed: false }), which produces strict MongoDB Extended JSON (e.g., numbers become $numberInt / $numberDouble). Calling this simply “JSON output” may confuse readers who try JSON.parse() and don’t get plain numbers. Consider explicitly noting that the output file is strict EJSON and should be parsed with EJSON.parse() (or switch to relaxed EJSON if plain JSON is desired).

The script writes report files according to `outputFormat` (`json`, `markdown`, or `both`) and also prints a console summary.

migration/toolbox/hotDocSpreadCheck/hot-doc-spread-check.json:5

• This file appears to be a captured output artifact from running hot-doc-spread-check.js (it contains generatedAt, stopReason, and sample hotDocuments). Because the script’s default outputFile is also hot-doc-spread-check.json, committing this will likely cause confusion and noisy diffs when users run the tool (they’ll overwrite a tracked file). Consider removing this from the repo, or renaming it to an explicit example fixture name (e.g. example-hot-doc-spread-check-output.ejson) and referencing it from the README.

{
  "generatedAt": "2026-08-18T11:14:15.032Z",
  "namespacesRequested": "ALL",
  "lookbackMs": {
    "$numberInt": "300000"

migration/toolbox/hotDocSpreadCheck/README.md:363

• The caught-up stop condition compares clusterTime to a locally captured timestamp; if the client clock is skewed relative to the cluster, the script may never report caught-up (or may do so early). It would help to document this assumption explicitly.

This issue also appears on line 373 of the same file.

  * it observed a matching event with `clusterTime` at or beyond the “now” watermark captured when the stream was opened

migration/toolbox/hotDocSpreadCheck/hot-doc-spread-check.js:199

• caught-up is detected by comparing evt.clusterTime (server logical time) to stopAtMs = Date.now() (client wall clock). If the client clock is skewed relative to the cluster, this comparison can be unreliable and may prevent caught-up from ever triggering (or trigger too early), changing stopReason/behavior on quiet workloads. Consider deriving the cutoff from a server-provided clusterTime/operationTime captured when opening the stream, rather than from local time.

    if (evt.clusterTime && evt.clusterTime.t * 1000 >= stopAtMs) {
      stopReason = "caught-up";
    }

[Copilot]

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

migration/toolbox/hotDocSpreadCheck/hot-doc-spread-check.js:276

• totalChangesPerSec / docChangesPerSec are computed using lookbackMs as the window duration even when the stream stops early with reasons like run-ms-exceeded or max-unique-docs-reached. In those stop cases the sample can be incomplete, so CPS-based gates can produce false negatives and the reported rates can be misleading. Consider explicitly flagging results as partial (based on stopReason) and/or computing rate denominators from the actual observed time span when not caught up.

  print(`Stream closed. Reason: ${stopReason}. Matched events: ${matchedEventsSeen}. Qualified doc events: ${qualifiedEventsSeen}.`);

  const windowSeconds = CFG.lookbackMs / 1000;
  const totalChangesPerSec = windowSeconds > 0 ? qualifiedEventsSeen / windowSeconds : 0;

[Copilot]

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (2)

migration/toolbox/hotDocSpreadCheck/README.md:6

• This tool relies on Change Streams, but the README doesn’t call out basic prerequisites (replica set/sharded cluster, connect to mongos for sharded, and appropriate privileges). Without that, users on unsupported topologies (e.g., standalone) will hit runtime failures without understanding why.

`hot-doc-spread-check.js` is a mongosh script that scans recent change activity and identifies documents that are hot enough to plausibly explain high applier skew.

migration/toolbox/hotDocSpreadCheck/README.md:556

• The Sources section links to a Google Doc, which is often access-restricted and can be a dead end for readers. Consider removing the hyperlink (or replacing it with a publicly accessible reference) since the README already includes the key derivation details.

## Sources

- [EP: Automatically detect and mitigate hot docs](https://docs.google.com/document/d/1mHBMjpeYnQKJyxAWjhUL7OBhGJ2733uakUWZZCsp5p4)

[sababich]

@niccottrell Hi Nicholas, I made multiple fixes to the script and README file to address all relevant findings of copilot review. I believe at this point the script is ready for production.