feat(helpers): streaming bulk-ingest with bounded memory

[margaretjgu]

Replaces the buffered readFileSync + splitIntoBatches approach with line-by-line streaming bounded to flush_bytes * concurrency bytes, so ingesting large files no longer requires loading the full dataset into memory. Concurrency is controlled with a semaphore that provides backpressure. NDJSON, JSON arrays, and CSV are all supported.

Closes #443

[github-actions]

✅MegaLinter analysis: Success

Descriptor	Linter	Files	Fixed	Errors	Warnings	Elapsed time
✅ COPYPASTE	jscpd	yes		no	no	1.03s
✅ REPOSITORY	gitleaks	yes		no	no	58.97s
✅ REPOSITORY	git_diff	yes		no	no	0.43s
✅ REPOSITORY	secretlint	yes		no	no	30.51s
✅ REPOSITORY	trivy	yes		no	no	16.4s
✅ TYPESCRIPT	eslint	4		0	0	4.13s

Notices

📣 MegaLinter 9.5.0 is out! Discover the new features and security recommendations in the release announcement. (Skip this info by defining SECURITY_SUGGESTIONS: false)

See detailed reports in MegaLinter artifacts
Set VALIDATE_ALL_CODEBASE: true in mega-linter.yml to validate all sources, not only the diff

MegaLinter is graciously provided by OX Security
Show us your support by starring ⭐ the repository

[JoshMock]

Very nice. Just one question but LGTM regardless. 👏

[JoshMock]

Nice. Very similar to how the bulk helper is implemented in the JS client. 👏

[JoshMock]

If this is released in a finally, is there any situation where one batch would completely fail to be ingested, but others would? If so, does the errors array capture enough detail to help identify what data has not been ingested? Or do we just assume they'd follow up with another bulk ingest that upserts to handle failures?

[margaretjgu]

thanks for catching this! i was focused on the happy path and missed that a fully failed batch wasn't being counted anywhere, so the summary would undercount... and only the first failed batch's error ever surfaced, so a second failure would've been silently lost.

fixed both: every batch's doc count gets counted now no matter what happens, and the error message lists every failed batch instead of just the first one.

[JoshMock]

A few things an agent review noted that might be worth running down. It went pretty hardcore so feel free to file some as follow-ups using your discretion.

• Uncaught crash on unreadable CSV source. stream.pipe(parser) does not forward source errors; a missing/unreadable file makes the ReadStream emit error with no listener → Unhandled 'error' event, process abort. Reproduced: error: ENOENT ... Emitted 'error' event on ReadStream instance (script over PR branch deps). Previously readFileSync ENOENT surfaced as input_error. fix: pipeline() from node:stream/promises, or stream.on('error', e => parser.destroy(e)).
• Error-code regression for non-CSV paths: file ENOENT/EACCES and malformed JSON-array SyntaxError no longer match the startsWith(...) allowlist in the handler, so they return transportError instead of input_error. Ingest failures caused by local input are now reported as transport failures.
• Empty-input regression: No input data received from file is gone. An empty --data-file, or empty piped stdin, now silently returns total: 0, succeeded: 0. Prior behavior was an explicit input_error.
• json source format is still fully buffered: arrayBuf accumulates the whole file before JSON.parse. Issue perf(es): bulk-ingest buffers all input into memory; OOMs on large dumps #443 acceptance explicitly requires all source formats to stream. Only NDJSON/CSV meet the goal; a multi-GB JSON array still OOMs. Not documented as a limitation anywhere
• source_format is now ignored for ndjson vs json; format is inferred from whether the first non-empty line starts with [. An explicit --source-format json on a file whose array starts mid-line, or --source-format ndjson on an array file, both silently do whatever detection decides. Prior code had the same auto-detect inside parseInput, so it is not new, but the flag is now decorative and worth deleting or honoring.
• No new tests for any of the new behavior: no streaming/bounded-memory test, no multi-batch ordering test, no error-path tests for missing file, malformed JSON array, or partial-batch failure. Deleted shared.test.ts coverage (parseInput line-number errors, buildBulkNdjsonBody action metadata) is not replaced by equivalent tests against the new inline logic.

[margaretjgu]

went through the rest of these.

fixed in this pr:

• csv missing/unreadable file was crashing the whole process (.pipe() doesn't forward source errors). now the stream error gets forwarded to the parser so it surfaces as input_error instead.
• ENOENT/EACCES/EISDIR on the ndjson/json path were falling through to transport_error. added them to the input_error classification.
• empty --data-file or empty piped stdin was silently returning total: 0 with no error. now throws input_error, matching the old behavior (an empty parsed array like [] still succeeds with 0 docs, that's not new and not a bug).

filed as a follow-up: #498, the --source-format flag being decorative for ndjson vs json. not new (old code auto-detected the same way) but worth deciding whether to honor it or drop the distinction.

added tests for all three fixes plus the missing-file/empty-file/header-only-csv edge cases.