[flyingdoubleG]

Summary

deepconf follows DeepConf (Fu et al., 2025) for token and group confidences and for the low/high percentile thresholds. Its final vote and some of its trace statistics depart from the paper and from the reference implementation (facebookresearch/deepconf, utils.py / wrapper.py). This PR brings them in line.

1. The vote barely counts agreement.
   • Each answer scored 0.7 · mean(confidence) + 0.3 · count / total.
   • Token confidences, −mean log P(top-k), are typically several units, while the frequency term is at most 1. The answer with the highest mean confidence therefore wins regardless of how many traces agree.
   • Example: 15 traces answering "42" at confidence 9.0 lose to a single trace answering "7" at 9.5.
   • DeepConf's confidence-weighted majority vote is V(a) = Σ_{t: answer_t = a} C_t, and the reference weighted_majority_vote sums the weights.
2. Early-terminated traces vote.
   • Traces stopped by the confidence threshold are unfinished, but they were included in the final vote. Their "answer" is whatever sentence they were cut off in.
   • The reference implementation skips them (stop_reason == 'gconf_threshold'). It also lets warmup traces vote only when their confidence reaches the threshold.
3. Short traces have zero group confidence.
   • Groups only form once a trace reaches window_size tokens (2048 by default). A shorter trace reports lowest_group = bottom_10_percent = 0.0.
   • The reference compute_least_grouped / calculate_bottom_window_confidence use the mean over the whole trace in that case.
4. The threshold was calibrated on a different statistic than the one it is compared against.
   • Online termination compares the current group confidence to the threshold.
   • The threshold was a percentile of the warmup traces' average confidence. DeepConf calibrates it on their lowest group confidence (min_confs in the reference).

Changes

• weighted_majority_vote: drops early-terminated traces and sums confidence_stats[confidence_metric] per answer. If every trace terminated early, it falls back to all of them.
• process_online: votes with the warmup traces that pass the threshold plus the online traces.
• ConfidenceCalculator: treats a trace shorter than the window as one group of all its tokens.
• Defaults: confidence_metric now defaults to "lowest_group". The README table is updated.

Tests

Three new tests in tests/test_deepconf.py, which CI runs: short-trace group confidence, sum-of-confidences voting, and skipping early-terminated traces. All three fail on main and pass with this change.