Repository navigation
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Index::update()deletes the index and dispatches oneInsertMultipleJobperchunk_sizedocuments. On the local driver each job'sinsertDocuments()reads, decodes, re-encodes and rewrites the whole index file, so a reindex of 10,000 entries at the default chunk size rewrites a file growing to 25 MB a hundred times. On a sync queue (the default install, andphp please search:updateon most sites) all of that happens inline in one process.Measured on a sync queue, median of 3 (scripts: https://gist.github.com/edalzell/06252c75c5c8d9bd14ab9f94a2041350).
update()is the full reindex including loading entries; "chunk replay" is only the file work the chunk jobs do; "single write" is the same documents written once.Fix:
Comb\Index::update()collects what the inline chunk jobs insert and writes the file once afterparent::update().IndexManagercaches index instances, so the sync job resolves the same instance that is mid-update. On a real queue the jobs run in other processes, the buffer stays empty and nothing changes; underBus::fake()the index is still deleted and nothing is written, so the existing dispatch tests pass untouched. Tests: 250 entries on a sync queue produce oneputwith all 250 documents; a faked bus writes nothing; inserts outsideupdate()still write immediately.