fix(tokenizer): never start a vocab load that can only be fetched

[zhanghanduo]

What

0.8.1 moved import tiktoken to the caller thread and added a 1 s atexit
join. That covers a cache hit, but not a cache miss: get_encoding() can perform
an unbounded network fetch plus BPE parsing, leaving its daemon thread alive when
the interpreter exits. Repeated cold-cache protocol tests reproduced one
post-response double free or corruption (fasttop) / -6 failure in 90 runs.

This PR starts no tokenizer thread unless the requested cl100k_base artifact is
already cached and passes tiktoken's expected SHA-256. Missing, disabled,
unrelated, corrupt, oversized, FIFO, and other non-regular cache
entries fail closed to the callers' CJK-aware heuristics. Operators can warm the
cache during build/setup, or explicitly restore runtime fetching with
AGENT_CORE_TIKTOKEN_FETCH=1 and accept the original exit-time risk.

Safety details

• The cache key and content hash are pinned for cl100k_base, the only encoding
  requested by AgentCore.
• Cache files are opened with non-blocking/no-follow flags where supported,
  accepted only when regular, and read at most 4 MiB before hashing. A FIFO or a
  concurrently growing file cannot turn validation into an unbounded read.
• The first caller claims an encoding before validation; concurrent callers
  return None instead of repeating the filesystem work.
• A real-tiktoken contract test compares the pinned URL/hash metadata and
  exercises tiktoken's actual cache lookup with network reads stubbed out. CI and
  release jobs run it in an isolated environment using the tokenizer version in
  uv.lock.

Consumer impact (MINOR)

A host with no valid warm cache now remains on approximate token counts instead
of fetching once and eventually becoming exact. The fallback is not a hard upper
bound: measured heuristic/real ratios were 0.84x for Chinese, 0.88x mixed
Chinese/Latin, 1.14x English prose, 0.72x JSON tool arguments, and roughly
0.10-0.125x for emoji-heavy samples. The context guard's 1.5x buffer does not
cover every input. Hosts sizing against a hard provider limit should warm the
cache.

Images that bake TIKTOKEN_CACHE_DIR remain on exact counts. If caching is
disabled with TIKTOKEN_CACHE_DIR="", set it to a writable directory before
running:

python -c "import tiktoken; tiktoken.get_encoding('cl100k_base')"

Tests that intentionally assert heuristic token arithmetic should pin that
estimator; this PR does so for the context-overflow regression test that was
previously sensitive to the host's cache state.

Verification

• Default environment: 1505 passed, 1 skipped
• Locked tokenizer extra with a real warm cache: 1506 passed
• Dedicated pin/cache-lookup contract test: passed
• Ruff: passed
• Pyright: 0 errors
• git diff --check: clean

Version remains 0.9.0: this deliberately changes observable cold-cache
behavior and therefore crosses the repository's 0.MINOR.PATCH compatibility
boundary.

[zhanghanduo]

Verification follow-up — two things the original description did not cover.

1. AgentCore's own venv has no tiktoken, so this repo's suite never exercises the real library; every tokenizer test runs against a fake loader. Checked the three real-library branches by shadow-loading this branch into a consumer venv that does have tiktoken:

arm	result
warm cache	threads=['tiktoken-init-cl100k_base'] — load runs, unchanged
cold cache	threads=[] + the WARNING (50/50 runs)
cold cache + AGENT_CORE_TIKTOKEN_FETCH=1	threads=[...] — old behaviour restored

The crash's precondition is what is gone: in the cold arm no tiktoken-init-* thread exists at all, so finalization has nothing to kill. Worth being explicit that counting clean runs would prove little here — the original failure rate was ~1%.

2. A consumer break, found by running the ApodexHarness suite against this branch with TIKTOKEN_CACHE_DIR pointed at an empty directory. tests/core/runtime/loop/test_tokenizer.py::test_first_call_is_nonblocking_even_if_load_is_slow monkeypatches a deliberately slow fake get_encoding and asserts the encoder eventually lands; the gate looks at the filesystem before sys.modules, so it now refuses a load that would never have touched the network. 1 failed, 8870 passed cold, and fully green warm (8871 passed; the one ERROR in both is a pre-existing env issue in that repo, unrelated). Now documented in the CHANGELOG with the fix for such tests: pin TIKTOKEN_CACHE_DIR at a directory holding any file, or set the opt-in env.

That also makes the point that this is invisible on a warm host — the same reason 0.8.1 looked complete.