Symptom
--source-format ndjson or --source-format json has no effect on how elastic es helpers bulk-ingest actually parses a file. The parser auto-detects a JSON array purely by checking whether the first non-empty line starts with [, regardless of what the flag says.
Repro: pass --source-format ndjson on a file whose contents are a JSON array. The array still gets parsed element by element, ignoring the flag.
This isn't new behavior, the old pre-streaming code auto-detected the same way inside parseByFormat, so this predates the recent streaming refactor. But now that the flag exists as documented input, it's worth either honoring it or removing the ndjson/json distinction from --source-format entirely (only csv actually changes behavior; --source-format still matters for --data-dir's default glob).
Where it happens
src/es/helpers/bulk-ingest.ts, the else branch in streamBulkIngest that sets isJsonArray from the first line instead of from opts.source_format.
Fix options
- Make
source_format authoritative: skip auto-detection, trust the flag.
- Or drop
ndjson/json as a meaningful distinction in --source-format (keep csv vs non-csv only) and document that array-vs-line format is auto-detected.
Flagged during review on #495.
Symptom
--source-format ndjsonor--source-format jsonhas no effect on howelastic es helpers bulk-ingestactually parses a file. The parser auto-detects a JSON array purely by checking whether the first non-empty line starts with[, regardless of what the flag says.Repro: pass
--source-format ndjsonon a file whose contents are a JSON array. The array still gets parsed element by element, ignoring the flag.This isn't new behavior, the old pre-streaming code auto-detected the same way inside
parseByFormat, so this predates the recent streaming refactor. But now that the flag exists as documented input, it's worth either honoring it or removing the ndjson/json distinction from--source-formatentirely (onlycsvactually changes behavior;--source-formatstill matters for--data-dir's default glob).Where it happens
src/es/helpers/bulk-ingest.ts, theelsebranch instreamBulkIngestthat setsisJsonArrayfrom the first line instead of fromopts.source_format.Fix options
source_formatauthoritative: skip auto-detection, trust the flag.ndjson/jsonas a meaningful distinction in--source-format(keepcsvvs non-csv only) and document that array-vs-line format is auto-detected.Flagged during review on #495.