[tomwangcn]

Summary

Fixes #683.

For models with a ugm vocabulary (e.g. BAAI/bge-m3), resolveBeginningTokenToPrepend() returned null unconditionally, so the BOS token was never prepended, even when the GGUF sets tokenizer.ggml.add_bos_token = true and model.tokens.shouldPrependBosToken is true. Since bge-m3 uses CLS pooling, the first content token was pooled instead of <s>, shifting embeddings away from the reference.

llama.cpp prepends BOS for UGM when add_bos is set (llama_vocab::impl::tokenize, case LLAMA_VOCAB_TYPE_UGM in llama-vocab.cpp: if (add_special && add_bos) output.push_back(special_bos_id);), and add_bos itself comes from the GGUF tokenizer.ggml.add_bos_token metadata (defaulting to false for UGM). The addon's shouldPrependBosToken() already exposes exactly that flag via llama_vocab_get_add_bos().

Changes

• src/utils/tokenizerUtils.ts: for ugm vocabularies, return tokens.bos when tokens.shouldPrependBosToken is true, and null otherwise. Models that don't request BOS (e.g. T5, whose GGUFs have add_bos_token = false) behave exactly as before. The ranking and completion paths share this helper, so they now match llama.cpp too.
• test/standalone/utils/tokenizerUtils.test.ts: new standalone tests covering beginning/end token resolution for ugm (with and without add_bos_token), spm, bpe, wpm, and rwkv.

Verification

• vitest run ./test/standalone/utils - 68 tests pass (12 new)
• tsc --noEmit --project tsconfig.json - clean
• eslint on the changed files - clean
• End-to-end against a real bge-m3 model file (gpustack/bge-m3-GGUF bge-m3-Q8_0.gguf, a Hugging Face mirror of BAAI/bge-m3; metadata read from the file: tokenizer.ggml.model = t5 (UGM), tokenizer.ggml.add_bos_token = true, tokenizer.ggml.bos_token_id = 0, bert.pooling_type = CLS):
  • resolveBeginningTokenToPrepend() returns null on master and 0 (<s>, the model's BOS) with this fix, so embedding inputs now start with <s>, matching the GGUF metadata and llama.cpp's own UGM path.
  • Measured 1024-dim embeddings of 5 varied inputs (English, Chinese, a single token, a longer paragraph) differ between master and this fix: cosine similarity 0.533-0.997, with short inputs shifting the most. This shows the missing BOS materially changes model outputs; I did not run a retrieval benchmark and did not compare against the native llama.cpp CLI binary.
• The reference tokenizer of BAAI/bge-m3 (tokenizer_config.json) is XLMRobertaTokenizer with bos_token = <s>, which prepends <s> by default - consistent with the GGUF's add_bos_token = true.