Repository navigation
Internal embedder with the default config only looks at the first 16 tokens of a chunk #359
Description
Which version of Nextcloud are you using?
35.0.0
Which version of PHP context_chat are you using?
5.5.0
Which version of backend context_chat are you using?
5.5.0
Nextcloud deployment method?
docker-compose
Describe the Bug
The default configs for CPU and GPU have batch_size set to 16 for the internal embedding model.
llama-cpp-python treats n_batch as the maximum length of each text and the inbuilt fastapi server from that package exposes the openai compatible api for embeddings at /v1/embeddings. It runs the embedding process with truncate=True by default (https://github.com/abetlen/llama-cpp-python/blob/1652066e0af45f2313b339670ef9555e8a54e545/llama_cpp/llama.py#L1071) which silently cuts off the parts that do not fit the batch_size in tokens, which is everything after 16 tokens in this case.
This is especially hard to detect since the output of embedding model is always a fixed 1024 sized list of numbers, and the quality is not that bad surprisingly considering only 16 tokens of a chunk got indexed. One thing to note here is that these 16 tokens were from each chunk and a document generally has multiple chunks so it still saw parts of the document.
Moreover, this affects the search part, in the answer generation phase, the LLM looks at the complete text of each chunk so this bug does not carry over to that part.
The fix should be to detect that the same multilingual-e5-large-instruct model is configured in an instance with batch size 16. If this is the case, we should show a setup check to the admin to perform a cleanup and re-index the documents.
We wouldn't do it automatically since the data is still not complete garbage and if an admin chooses to continue with the current state, they can.
This does not apply to setups where:
- a remote embedding service is setup
- batch size is not 16
- a different embedding model is configured
To Reproduce
- Have two text files
first-file.txtandsecond-file.txtwith the following contents:
first-file.txt:
Air India is the flag carrier of India, headquartered in Gurugram, Haryana. The airline is owned by Air India Limited, which is owned by the Tata Group (74.9%) and Singapore Airlines (25.1%).
My name is Koala.
second-file.txt
Air India is the flag carrier of India, headquartered in Gurugram, Haryana. The airline is owned by Air India Limited, which is owned by the Tata Group (74.9%) and Singapore Airlines (25.1%).
My mom is a scientist in sanpanpisko.
- Upload them to nextcloud or create files using the text app.
- Check the logs to ensure two sources and two chunks were embedded. Here the content is such that it fits in the 2000 character limit of a chunk so has only one chunk for the whole content, and the first 16 tokens are exactly the same in both so the embedding from both come out to be the same.
- Check the vector db for the two docs.
ccb=# select * from docs where source_id in ('files__default: 24175', 'files__default: 24176');
source_id | provider | modified | chunks
-----------------------+----------------+---------------------+----------------------------------------
files__default: 24175 | files__default | 2026-10-15 13:20:19 | {c79e1ea2-1590-47c8-ad7e-6abe55fdb1e9}
files__default: 24176 | files__default | 2026-10-15 13:19:44 | {132eb330-d4dc-4818-977c-5c64c2da675f}
(2 rows)
select embedding from langchain_pg_embedding where id in ('c79e1ea2-1590-47c8-ad7e-6abe55fdb1e9', '132eb330-d4dc-4818-977c-5c64c2da675f');
...
[0.6259186,0.5404417,-0.6474465,-1.3302455,0.25856087,-0.52309173, ...]
[0.6259186,0.5404417,-0.6474465,-1.3302455,0.25856087,-0.52309173, ...]
PHP logs (Warning these might contain sensitive information)
No response
Ex-App logs (Warning these might contain sensitive information)
No response
Server logs (if applicable)
No response
Activity
marcelklehr commentedon Oct 21, 2026
Mmh, why would they though?
kyteinsky commentedon Oct 21, 2026
since the data is not completely wrong, and it's just less accurate results, like with our CI, it can find the documents with a 30 chunk limit.
but yeah ideally, the files should be reindexed.
Aniruthan-0709 commentedon Oct 25, 2026
Hi, I'd like to work on this if nobody else has started on it.
My first step is to reproduce it on the default CPU config: embed the two files from the description and confirm that both chunks get identical vectors with
n_batch: 16, then check that they differ once the batch size is large enough for the whole chunk.On main, config.cpu.yaml and config.gpu.yaml both set n_batch: 16 for the multilingual-e5-large-instruct embedder (the batch_size key there is commented out and only applies to a remote service). One question before I go further: would a backend-only change be enough here? I'm thinking of fixing the default
n_batchinconfig.cpu.yamlandconfig.gpu.yamlso new installs embed full chunks, plus a way for the backend to report that an existing instance indexed with the old setting needs re-indexing. The admin setup check would then live in the PHP app.I'll post what I find from the repro by Friday, October 30.
kyteinsky commentedon Oct 26, 2026
hello, thanks for your enthusiasm, yes the general approach is good, we also need to fix the existing installs with a wipe of the vector db + a reindex.
given the sensitive nature of this issue, we would like to do it ourselves.
but feel free to pick from one of the other existing issues.
kyteinsky commentedon Oct 26, 2026
@marcelklehr
should we have an event/function call for "we're nuking everything" for the content providers? We have the intial index one.
in the context of doing a fresh reindex, we remove every vector from the vector DB, clear out all queues in the php side and then trigger initial import for files and for all the content providers so maybe makes sense to trigger a db cleared function so the content providers can flush their queues, or is it fine without it?
marcelklehr commentedon Oct 26, 2026
Checking both mail and zammad...
other apps to check/inform:
marcelklehr commentedon Oct 26, 2026
I'll send PRs for these to listen to initialImport
kyteinsky commentedon Oct 26, 2026
mail may not be fine :/
https://github.com/nextcloud/mail/blob/2f5819a313f9ffc796a853f38ff8c1cc1810b707/lib/ContextChat/ContextChatProvider.php#L126-L127
this would need to be reset on initial index trigger, afaict: https://github.com/nextcloud/mail/blob/main/lib/Db/ContextChat/Task.php
and bg job should be rescheduled: https://github.com/nextcloud/mail/blob/main/lib/BackgroundJob/ContextChat/ScheduleJob.php
kyteinsky commentedon Oct 26, 2026
nice thanks!