Add multi-turn cache affinity for sampling

[kailash109]

Summary

• add an optional cache_affinity_key that keeps turns from one trajectory on the same Modal replica and DP rank
• preserve legacy per-request routing when the key is omitted, with validation and idempotency coverage
• add configurable local and Modal ablation harnesses that report cold-inclusive cache metrics, reusable-prefix metrics, and end-to-end timing
• support configurable H100/H200 execution, replica counts, CUDA graphs, and fixed-length generation via ignore_eos

8x H100 fixed-length validation

Eight one-H100 replicas running Qwen3-8B with CUDA graphs, 4 trajectories, 8 turns, 4,096-token initial prompts, exactly 512 generated tokens per request, and a 16K context. Both arms perform exactly 32 requests and decode 16,384 tokens. The harness verifies all eight distinct replicas and their SGLang backends are healthy before measurement.

Across each trajectory, generated history grows the reusable prefix by 3,584 tokens:

• cold-inclusive input cache hit rate: 91.29% with affinity vs 25.54% without (+65.75 points)
• reusable-prefix hit rate: 100% with affinity vs 27.98% without
• full reusable-prefix hits: 28/28 with affinity vs 4/28 without
• total request wall time: 140.79s with affinity vs 148.51s without (1.055x speedup, 5.2% lower)
• mean warm-turn latency: 4.403s with affinity vs 4.669s without (1.061x speedup, 5.7% lower)
• end-to-end p95 latency: 4.610s with affinity vs 5.053s without
• complete Modal invocation including eight model loads, CUDA graph capture, both arms, and shutdown: 8m37s

Modal run: https://modal.com/apps/modal-labs/kailash-dev/ap-IibK6PzfrLcclpQV6ZKlfO

The equal-work result is decode-bound: decoding 512 tokens takes approximately 4.4s on Qwen3-8B/H100, while avoiding repeated prefill saves approximately 0.27s per warm request on average.

Test plan

• uv run pytest tests/inference/test_sampling.py tests/control_plane/test_sampling.py tests/control_plane/test_http.py
• uv run ruff check src/lilo/control_plane/http.py src/lilo/inference/sampling.py tests/control_plane/test_http.py tests/control_plane/test_sampling.py tests/inference/test_sampling.py scripts/validate_multiturn_cache_affinity.py scripts/validate_multiturn_cache_affinity_modal.py
• Qwen3-8B, exact 512-output-token, 8-turn, eight-H100 Modal benchmark in kailash-dev
• verified all benchmark H100 containers stopped after completion