Skip to content

fix: set source_id on RecursiveDocumentSplitter chunks#12155

Open
onatozmenn wants to merge 1 commit into
deepset-ai:mainfrom
onatozmenn:fix/recursive-splitter-source-id
Open

fix: set source_id on RecursiveDocumentSplitter chunks#12155
onatozmenn wants to merge 1 commit into
deepset-ai:mainfrom
onatozmenn:fix/recursive-splitter-source-id

Conversation

@onatozmenn

Copy link
Copy Markdown

Related Issues

Proposed Changes:

RecursiveDocumentSplitter wrote only parent_id on its chunks. Every other splitter writes source_id, and that is the key the rest of Haystack reads to tie a chunk back to its document, so the output did not compose with components that follow the convention. SentenceWindowRetriever reads source_id by default and raises when it is absent, so the same pipeline worked with DocumentSplitter and failed with this one.

Chunks now set source_id alongside parent_id. parent_id keeps its previous value, so anything already reading it is unaffected, and this is additive for anyone reading the meta dict.

I went with adding the key rather than renaming it, or teaching the retriever about parent_id. Renaming would break existing readers, and teaching one consumer about the alias would leave the underlying inconsistency in place for the next one.

The class docstring also needed a correction. It showed a meta key named original_id, which the code has never written, so the documented key, the emitted key and the key consumers read were three different names. Its output block did not match the example either: it was generated with a different split_length than the one in the snippet (260 produces a single chunk, the four shown come out at 15) and reported _split_overlap as [] where a run with split_overlap=0 gives None. I regenerated the block by running the example so it now reproduces what it claims.

How did you test it?

Two new unit tests in test/components/preprocessors/test_recursive_splitter.py:

  • test_recursive_splitter_sets_source_id — chunks carry source_id, and parent_id still holds the same value.
  • test_recursive_splitter_output_works_with_sentence_window_retriever — the actual reported failure, end to end through InMemoryDocumentStore and SentenceWindowRetriever.

Both fail on main, with KeyError: 'source_id' and ValueError: The retrieved documents must have 'source_id' in their metadata. respectively.

hatch run test:unit test/components/preprocessors/ test/components/retrievers/
526 passed, 49 deselected

hatch run fmt-check reports the same two PLR0917 errors before and after my change, both in azure_document_embedder.py and openai_document_embedder.py, which this PR does not touch.

Notes for the reviewer

The interesting line is the one added in _run_one. Everything else is the docstring and the tests.

Worth deciding explicitly: parent_id is now redundant. I kept it because removing it would be a breaking change and is not needed to fix the issue, but if you would rather deprecate it, that is a natural follow-up and I am happy to do it.

HierarchicalDocumentSplitter already emits both keys, so this brings RecursiveDocumentSplitter in line with it.

Checklist

I used AI assistance while investigating this and preparing the change. I reviewed and verified it myself.

RecursiveDocumentSplitter wrote only `parent_id` on the chunks it produced.
Every other splitter in the library writes `source_id`, and that is the key the
rest of Haystack reads to tie a chunk back to the document it came from.

The practical effect is that its output does not compose with
SentenceWindowRetriever, which reads `source_id` by default and raises when it
is missing. The same pipeline works with DocumentSplitter and fails with
RecursiveDocumentSplitter:

    DocumentSplitter          -> OK, 3 context documents
    RecursiveDocumentSplitter -> ValueError: The retrieved documents must have
                                 'source_id' in their metadata.

Chunks now carry `source_id` as well as `parent_id`, which keeps its previous
value so callers already reading it are unaffected.

While here, the class docstring example claimed a meta key named `original_id`,
which the code has never written. Its output block also did not match what the
example produces: it was generated with a different `split_length` than the one
shown, and reported `_split_overlap` as `[]` where a run with `split_overlap=0`
gives `None`. The example now reproduces the output it claims.
@onatozmenn
onatozmenn requested a review from a team as a code owner July 25, 2026 09:00
@onatozmenn
onatozmenn requested review from sjrl and removed request for a team July 25, 2026 09:00
@vercel

vercel Bot commented Jul 25, 2026

Copy link
Copy Markdown

@onatozmenn is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Jul 25, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RecursiveDocumentSplitter does not set source_id, so its chunks break SentenceWindowRetriever

2 participants