fix: set source_id on RecursiveDocumentSplitter chunks#12155
Open
onatozmenn wants to merge 1 commit into
Open
Conversation
RecursiveDocumentSplitter wrote only `parent_id` on the chunks it produced.
Every other splitter in the library writes `source_id`, and that is the key the
rest of Haystack reads to tie a chunk back to the document it came from.
The practical effect is that its output does not compose with
SentenceWindowRetriever, which reads `source_id` by default and raises when it
is missing. The same pipeline works with DocumentSplitter and fails with
RecursiveDocumentSplitter:
DocumentSplitter -> OK, 3 context documents
RecursiveDocumentSplitter -> ValueError: The retrieved documents must have
'source_id' in their metadata.
Chunks now carry `source_id` as well as `parent_id`, which keeps its previous
value so callers already reading it are unaffected.
While here, the class docstring example claimed a meta key named `original_id`,
which the code has never written. Its output block also did not match what the
example produces: it was generated with a different `split_length` than the one
shown, and reported `_split_overlap` as `[]` where a run with `split_overlap=0`
gives `None`. The example now reproduces the output it claims.
|
@onatozmenn is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues
Proposed Changes:
RecursiveDocumentSplitterwrote onlyparent_idon its chunks. Every other splitter writessource_id, and that is the key the rest of Haystack reads to tie a chunk back to its document, so the output did not compose with components that follow the convention.SentenceWindowRetrieverreadssource_idby default and raises when it is absent, so the same pipeline worked withDocumentSplitterand failed with this one.Chunks now set
source_idalongsideparent_id.parent_idkeeps its previous value, so anything already reading it is unaffected, and this is additive for anyone reading the meta dict.I went with adding the key rather than renaming it, or teaching the retriever about
parent_id. Renaming would break existing readers, and teaching one consumer about the alias would leave the underlying inconsistency in place for the next one.The class docstring also needed a correction. It showed a meta key named
original_id, which the code has never written, so the documented key, the emitted key and the key consumers read were three different names. Its output block did not match the example either: it was generated with a differentsplit_lengththan the one in the snippet (260 produces a single chunk, the four shown come out at 15) and reported_split_overlapas[]where a run withsplit_overlap=0givesNone. I regenerated the block by running the example so it now reproduces what it claims.How did you test it?
Two new unit tests in
test/components/preprocessors/test_recursive_splitter.py:test_recursive_splitter_sets_source_id— chunks carrysource_id, andparent_idstill holds the same value.test_recursive_splitter_output_works_with_sentence_window_retriever— the actual reported failure, end to end throughInMemoryDocumentStoreandSentenceWindowRetriever.Both fail on
main, withKeyError: 'source_id'andValueError: The retrieved documents must have 'source_id' in their metadata.respectively.hatch run fmt-checkreports the same twoPLR0917errors before and after my change, both inazure_document_embedder.pyandopenai_document_embedder.py, which this PR does not touch.Notes for the reviewer
The interesting line is the one added in
_run_one. Everything else is the docstring and the tests.Worth deciding explicitly:
parent_idis now redundant. I kept it because removing it would be a breaking change and is not needed to fix the issue, but if you would rather deprecate it, that is a natural follow-up and I am happy to do it.HierarchicalDocumentSplitteralready emits both keys, so this bringsRecursiveDocumentSplitterin line with it.Checklist
fix:.I used AI assistance while investigating this and preparing the change. I reviewed and verified it myself.