[flyingdoubleG]

Summary

entropy_decoding._sample (used by decoding="entropy_decoding" for local inference) has two filtering bugs.

1. Top-p removes the head of the distribution instead of the tail.
   • The top-k probabilities are flipped into ascending order before cumsum. The mask probs_sum - probs_sort > top_p therefore drops every token whose less likely predecessors already exceed top_p, i.e. the most likely tokens.
   • Example: one token at p=0.05 and 39 at p=0.0244, with top_p=0.9 and top_k=40. The most likely token is never sampled, the three largest are removed, and the whole tail stays.
2. min_p is a no-op.
   • It masks logit, but sampling continues from probs, which were computed before the mask.
   • Example: with min_p=0.1 and p = [0.5, 0.3, 20 × 0.01], tokens below 0.05 are still sampled about 20% of the time.

Changes

• Apply min_p to the probabilities.
• Keep the top-k probabilities in descending order for the nucleus test, so it always keeps the most likely token and drops the least likely tail.
• This also removes the two torch.tensor(..., device=device) constants in the mask, which used the module's global device rather than the logits' device.

Tests

New tests/test_entropy_decoding_sampling.py checks:

• the set of sampled tokens under top-p, which should be the first 36 of 40 in the example above;
• that min_p removes the tail.

Both tests fail on main and pass with this change.