Rogue SPR: score single-leaf candidates from parent split overlaps

.AGENTS/memory/Optimizations.md

@@ -987,3 +987,36 @@ row-filling skips ≤1.5% of fill.  Next lever: build child cost matrices
 incrementally from the parent (97.6% of candidates drop one leaf); the fill is
 now 37–55% of runtime.  Parent prices are computed lazily (first candidate
 past the row bound).
+
+## Rogue SPR: single-leaf candidates from parent overlaps (2026-10-19)
+Measured first (rdtsc-instrumented release build, workloads in
+`dev/rogue_bound/workloads.rds`): fill 35% of runtime at 80 tips, 54% at 150;
+KeepAndReduce + GetSplits + FirstMatchingSplit 46% / 31%.  **Popcount is not
+the fill's cost**: in a standalone kernel (1-3 64-bit bins), replacing the
+popcount with a lookup of parent counts saves 1 of ~14.5 cycles/cell.  The
+cost is the per-cell log arithmetic.  So "O(1) quadrant counts" alone does not
+pay; what pays is what the parent overlaps let a candidate *skip*.
+- Tier 1 (all bounded MCI calls): cell information as
+  `sum_q f(n_q) + row_term + col_term`, `f(k) = k lg2 k` tabled
+  (`KLg2K()`), row/column terms hoisted.  Kernel 14.5 -> 8.5 cycles/cell;
+  costs agree within 1 unit.  Alone: -6..-16% (80/150 tips).
+- Tier 2 (`SingleDrop` in `spr.cpp`, `mutual_clustering_bounded_counts()`):
+  for leaf l != 1 whose removal neither creates a reducible structure
+  (cherry/root/chain tests of `reduce_trees()` replayed on ledgers with l
+  removed) nor makes a pair of splits identical (precomputed: parent pairs
+  differing in l alone), the child matrix comes from parent overlaps minus l's
+  membership; the vanishing split per tree is par(l)'s row, or sib(l)'s if
+  par(l) is the root's child.  96% of candidates qualify (84% on 15-move
+  pairs).  Predicate agreed with KeepAndReduce on all 589,686 candidates.
+- Min of 9 interleaved runs vs main: rand30 -56%, rand80 -61%, rand150 -60%,
+  mid80 -53%, mid150 -55%; depth 2 only -3% (pair candidates still rebuild).
+- Correctness: 13,833 score vectors (2.9 M candidates): argmin sets identical
+  to the non-incremental path in the same build; vs main, 5 differ, all exact
+  ties (within 1e-12) that main broke by rounding noise.  1/410 SPRDist
+  results changed (an 8-leaf tie).  Max score difference 4e-13 bits.
+- Dropped from the counts path: the initial entropy bound (never fired in
+  600k candidates).  Rejected: storing parent cell information and
+  updating it by an O(1) delta (5.3 cycles/cell vs 8.5) - not worth a d^2
+  double matrix once the fill is no longer dominant.
+- Next: depth-2 pair candidates (two leaves) by the same route; the
+  unbounded MCI could use the Tier 1 form if tests tolerate ulp changes.

NEWS.md

@@ -18,6 +18,13 @@
   problems and warm-starting the rest: c. 50% faster at 80 leaves and 65% at
   150 leaves on unrelated trees.
 
+- `SPRDist(method = "Rogue")` scores most single-leaf candidates from the
+  overlaps between the parent trees' splits, without rebuilding, reducing or
+  splitting the smaller trees, and fills cost matrices with fewer operations:
+  a further 55–60% faster at 30–150 leaves.  Scores can differ from before in
+  the last few bits, which can break exact ties between candidates
+  differently.
+
 - Improve performance when comparing many similar trees with
   `ClusteringInfoDistance()`, `MatchingSplitDistance()`, `InfoRobinsonFoulds()`
   and `JaccardRobinsonFoulds()`.

R/RcppExports.R

@@ -208,8 +208,8 @@ keep_and_reduce <- function(tree1, tree2, keep) {
     .Call(`_TreeDist_keep_and_reduce`, tree1, tree2, keep)
 }
 
-spr_rogue_scores <- function(tree1, tree2, depth) {
-    .Call(`_TreeDist_spr_rogue_scores`, tree1, tree2, depth)
+spr_rogue_scores <- function(tree1, tree2, depth, incremental = TRUE) {
+    .Call(`_TreeDist_spr_rogue_scores`, tree1, tree2, depth, incremental)
 }
 
 spr_bounded_cid <- function(tree1, tree2, cutoff) {

dev/rogue_mci_bound_findings.md

@@ -72,3 +72,7 @@ The LAPs that remain run 2.0×, 3.4× and 4.1× faster at 30, 80 and 150 tips. T
    - 97.6% of candidates drop a single leaf. A child cell's quadrant counts are the parent cell's counts minus that leaf's quadrant, so each cell costs O(1).
    - This would also skip KeepAndReduce and GetSplits (about 30% of runtime) for those candidates.
    - It is the largest remaining lever.
+
+## Follow-up: single-leaf candidates (2026-10-19)
+
+Done; see `.AGENTS/memory/Optimizations.md`, "Rogue SPR: single-leaf candidates from parent overlaps". Popcount turned out not to be the fill's cost (parent counts save ~1 of ~14.5 cycles per cell); the gains come from a cheaper cell formula and from skipping KeepAndReduce, GetSplits and FirstMatchingSplit for the 96% of candidates whose trees need no further reduction: 55–61% faster at 30–150 leaves.

src/RcppExports.cpp

@@ -546,15 +546,16 @@ BEGIN_RCPP
 END_RCPP
 }
 // spr_rogue_scores
-List spr_rogue_scores(const List& tree1, const List& tree2, const int depth);
-RcppExport SEXP _TreeDist_spr_rogue_scores(SEXP tree1SEXP, SEXP tree2SEXP, SEXP depthSEXP) {
+List spr_rogue_scores(const List& tree1, const List& tree2, const int depth, const bool incremental);
+RcppExport SEXP _TreeDist_spr_rogue_scores(SEXP tree1SEXP, SEXP tree2SEXP, SEXP depthSEXP, SEXP incrementalSEXP) {
 BEGIN_RCPP
     Rcpp::RObject rcpp_result_gen;
     Rcpp::RNGScope rcpp_rngScope_gen;
     Rcpp::traits::input_parameter< const List& >::type tree1(tree1SEXP);
     Rcpp::traits::input_parameter< const List& >::type tree2(tree2SEXP);
     Rcpp::traits::input_parameter< const int >::type depth(depthSEXP);
-    rcpp_result_gen = Rcpp::wrap(spr_rogue_scores(tree1, tree2, depth));
+    Rcpp::traits::input_parameter< const bool >::type incremental(incrementalSEXP);
+    rcpp_result_gen = Rcpp::wrap(spr_rogue_scores(tree1, tree2, depth, incremental));
     return rcpp_result_gen;
 END_RCPP
 }
@@ -828,7 +829,6 @@ BEGIN_RCPP
     return rcpp_result_gen;
 END_RCPP
 }
-
 // cpp_popcount_force_sw
 bool cpp_popcount_force_sw(const bool force);
 RcppExport SEXP _TreeDist_cpp_popcount_force_sw(SEXP forceSEXP) {
@@ -885,7 +885,7 @@ static const R_CallMethodDef CallEntries[] = {
     {"_TreeDist_confusion", (DL_FUNC) &_TreeDist_confusion, 2},
     {"_TreeDist_keep_and_reroot", (DL_FUNC) &_TreeDist_keep_and_reroot, 3},
     {"_TreeDist_keep_and_reduce", (DL_FUNC) &_TreeDist_keep_and_reduce, 3},
-    {"_TreeDist_spr_rogue_scores", (DL_FUNC) &_TreeDist_spr_rogue_scores, 3},
+    {"_TreeDist_spr_rogue_scores", (DL_FUNC) &_TreeDist_spr_rogue_scores, 4},
     {"_TreeDist_spr_bounded_cid", (DL_FUNC) &_TreeDist_spr_bounded_cid, 3},
     {"_TreeDist_spr_first_shared_split", (DL_FUNC) &_TreeDist_spr_first_shared_split, 2},
     {"_TreeDist_spr_keep_and_reduce", (DL_FUNC) &_TreeDist_spr_keep_and_reduce, 3},

src/split_counts.h

@@ -0,0 +1,27 @@
+#ifndef TREEDIST_SPLIT_COUNTS_H_
+#define TREEDIST_SPLIT_COUNTS_H_
+
+#include <cstdint>
+
+// Splits of a pair of trees, each obtained from a parent tree by removing
+// one leaf, described through the parent's splits.  The trees have `n_split`
+// splits each.  Child split i of the first tree restricts parent split
+// row[i]; it has row_in[i] leaves, and row_has[i] is 1 if the parent split
+// contained the removed leaf, else 0.  col, col_in and col_has describe the
+// second tree likewise.  overlap[r * stride + c] counts the leaves shared by
+// parent splits r and c, so the corresponding child splits share
+//   overlap[r * stride + c] - (row_has & col_has).
+struct SplitCounts {
+  int n_tip; // Leaves in each child tree
+  int n_split;
+  const int16_t *overlap;
+  int stride;
+  const int *row;
+  const int *row_in;
+  const int16_t *row_has;
+  const int *col;
+  const int *col_in;
+  const int16_t *col_has;
+};
+
+#endif