feat(engine): add BPE tokenizer and offline asset tooling

[chamsechan]

Adds a shared BPE tokenizer and deterministic .eftok asset tooling for the supported Qwen2.5 and Gemma3 tokenizer pipelines. Tokenization remains an internal Model Execution helper, with immutable mapped assets, Unicode normalization, byte-level pretokenization, added-token handling, and HF-compatible decoding.

The compiler preserves merge ranks, or uses compact score data only when it verifies an equivalent total merge order. It rejects unsupported components and incomplete base-vocabulary byte fallback. Differential verification rejects missing output, and loading tests isolate temporary files for parallel execution.

This delivers S1 only. The plan records the later tensor-graph LLM work; that Model and its end-to-end configurations are not introduced here. The documented NFC complexity matches the implementation's O(n log n) sorting.

Validation:

• Canonical local gate: 429 registered tests, 426 passed and 3 optional-asset skips; rerun by the delivery script before committing.
• Focused BPE tests with real assets: 28/28 passed; ASan/UBSan parallel run: 28/28 passed.
• HF differential: 1,866 encoding cases per model, plus 3,800 Qwen and 6,800 Gemma decoding cases, all matching.
• Regenerated both tokenizer assets and confirmed their bytes and SHA-256 values match the manifest.
• Independent review confirmed the score-order check, output completeness, byte fallback, test isolation, and long NFC input fixes.

The two BPE tests skipped by the default gate were run with real assets in the focused and sanitizer checks. The optional real GGUF test remains skipped because its asset was not supplied.