Skip to content

[6.x] Write the Comb index once when updating on a sync queue - #15726

Open
edalzell wants to merge 2 commits into
statamic:6.xfrom
edalzell:fix/comb-reindex-rewrites-file-per-chunk
Open

edalzell wants to merge 2 commits into
statamic:6.xfrom
edalzell:fix/comb-reindex-rewrites-file-per-chunk

Conversation

@edalzell

Copy link
Copy Markdown
Contributor

Index::update() deletes the index and dispatches one InsertMultipleJob per chunk_size documents. On the local driver each job's insertDocuments() reads, decodes, re-encodes and rewrites the whole index file, so a reindex of 10,000 entries at the default chunk size rewrites a file growing to 25 MB a hundred times. On a sync queue (the default install, and php please search:update on most sites) all of that happens inline in one process.

Measured on a sync queue, median of 3 (scripts: https://gist.github.com/edalzell/06252c75c5c8d9bd14ab9f94a2041350). update() is the full reindex including loading entries; "chunk replay" is only the file work the chunk jobs do; "single write" is the same documents written once.

docs file update() before update() after chunk replay single write
2,001 5 MB 333 ms 213 ms 142 ms 4 ms
10,000 25 MB 4,650 ms 1,183 ms 3,500 ms 22 ms

Fix: Comb\Index::update() collects what the inline chunk jobs insert and writes the file once after parent::update(). IndexManager caches index instances, so the sync job resolves the same instance that is mid-update. On a real queue the jobs run in other processes, the buffer stays empty and nothing changes; under Bus::fake() the index is still deleted and nothing is written, so the existing dispatch tests pass untouched. Tests: 250 entries on a sync queue produce one put with all 250 documents; a faked bus writes nothing; inserts outside update() still write immediately.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant