Skip to content

Prune trees with different leaf sets in C++ (#123) - #221

Open
ms609-agent wants to merge 9 commits into
mainfrom
claude/treedist-issue-123-4b91f7
Open

ms609-agent wants to merge 9 commits into
mainfrom
claude/treedist-issue-123-4b91f7

Conversation

@ms609-agent

Copy link
Copy Markdown
Collaborator

Closes #123.

What

Trees with different leaf sets were compared by pruning each pair with KeepTip() in R — once for the score (.SplitDistanceAllPairs() etc.), again in .MaxValue(), and again in NormalizeInfo() — holding n(n−1) pruned phylo objects in memory at once.

New cpp_pruned_pairs() (in pairwise_distances.cpp) restricts each tree's splits to the leaves it shares with its partner, drops trivial splits, merges splits that coincide after pruning, then scores the pair with the existing batch kernels and reports each pruned tree's information content (clustering entropy / splitwise info / split count). It runs in OpenMP over pairs, with per-thread scratch. Wired into:

  • the different-leaves branches of .SplitDistanceAllPairs(), .SplitDistanceManyMany() and .SplitDistanceOneMany() (MCI, IRF, MSD, MSI, SPI, Nye/Jaccard, RF);
  • .MaxValue() and NormalizeInfo() when the information function is ClusteringEntropy, SplitwiseInfo or NSplits. NormalizeInfo() also no longer prunes trees when infoInBoth or a numeric how is given.

Other scoring and information functions keep the R path.

Benchmark (150 random trees, 40–70 of 80 leaves, 1 thread, optimised build)

before after
TreeDistance(trees) 15.6 s 1.0 s
PhylogeneticInfoDistance(trees, normalize = TRUE) 19.3 s 0.9 s
RobinsonFoulds(trees[1:60], trees[61:150]) 2.8 s 0.13 s
peak memory (gc() max, MB) 104 67

Bug fixes found along the way (in NEWS)

  1. All-pairs values were wrong when leaf sets differed: KeepTip() on a Splits object keeps that tree's own leaf order, so the two pruned trees' bits could refer to different leaves. Example: MCI 2.60 from all-pairs vs 3.56 from comparing the pruned pair directly. The new code matches the KeepTip()-on-phylo reference for every metric (see test-pruned_pairs.R).
  2. RobinsonFoulds(trees) with differing leaf sets went through the ClusterTable path, which ignores the leaf differences, and returned wrong values.
  3. f(list1, list2) was transposed when leaf sets differed (rows were tree2), unlike the same-leaves case. Rows now always correspond to tree1. .SharedOnly() now lists N × M pairs in column-major order to match. One existing expectation in test-different-tips.R encoded the transposed shape and is updated. This is a user-visible change.

Notes for review

  • SplitList can only be built from an RawMatrix, so each thread builds one before the parallel region, sized to the largest input, and overwrites its public fields for each pair. That depends on the TreeTools SplitList layout (row stride ≥ pruned bin count); a capacity constructor in TreeTools would be tidier but would need a LinkingTo bump.
  • Pairs with fewer than four shared leaves score 0 with 0 information, so normalized distances are NaN, which matches the existing 1-vs-1 behaviour.
  • I synthesised a test set rather than downloading the reporter's attachment.
  • spelling::spell_check_package() errors locally on a Windows UTF-8 issue that already exists in NEWS.md; CI's tests/spelling.R will cover it.

🤖 Generated with Claude Code

Pairs of trees with different leaves were compared by pruning each pair
with KeepTip() in R, once for the score and again for each normalizing
value, holding n(n-1) pruned trees in memory.  cpp_pruned_pairs() now
restricts each tree's splits to the shared leaves, merges splits that
coincide, and scores the pair and its information content in one OpenMP
pass.  TreeDistance() on 150 gene trees: 15.6 s -> 1.0 s.

Also fixes, for trees with differing leaf sets:
- all-pairs scores, which misaligned leaves when pruning Splits objects;
- RobinsonFoulds(trees), which ignored the differing leaves;
- tree1-vs-tree2 matrices, which were transposed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ms609-agent
ms609-agent requested a review from ms609 October 19, 2026 09:50
codemeta-writer Bot and others added 2 commits October 19, 2026 09:50
NEWS.md had a cp1252 multiplication sign, which spell_check_package()
rejects as invalid UTF-8.

The r-hub fedora-44 ape binary links libRlapack.so, which the current
gcc-asan image no longer ships, so dependency installation failed before
any TreeDist code ran.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

This comment has been minimized.

ms609-agent and others added 2 commits October 19, 2026 11:42
Building ape from source fixed only ape: mvtnorm and other r-hub
fedora-44 binaries also link libRlapack.so, which the gcc-asan image
lacks.  Symlink it to the LAPACK library R itself uses instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

This comment has been minimized.

@codecov

codecov Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.65%. Comparing base (9f3455f) to head (c3b2464).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #221      +/-   ##
==========================================
+ Coverage   97.57%   97.65%   +0.07%     
==========================================
  Files          60       60              
  Lines        6056     6305     +249     
==========================================
+ Hits         5909     6157     +248     
- Misses        147      148       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Checking leaf sets with TipLabels() slowed RobinsonFoulds(trees) by c. 8%
(CI benchmark).  as.ClusterTable() already errors on a leaf missing from
the first tree; trees whose leaves are a strict subset are caught by
comparing each table's leaf count, in C++.  The same check closes that
loophole in the cross-pairs fast path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

This comment has been minimized.

ms609-agent and others added 2 commits October 19, 2026 13:06
The R loop used when a scoring function has no C++ version (or a cluster
is registered) still pruned Splits without reordering their leaves.
Cover it and the one-vs-many fallback with tests, restoring project
coverage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
readxl 1.6.0 needs R >= 4.2; the 2026-06-05 snapshot has 1.5.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

This comment has been minimized.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Performance benchmark results

Call Status Change Time (ms)
ClusteringInfoDistance(tr280) ⚪ NSD -1.32% 9.11, 9.23 →
9.39, 9.29
ClusteringInfoDistance(tr50) ⚪ NSD -0.37% 6.56, 6.68 →
6.66, 6.66
LAPJV(test2000) ⚪ NSD -1.64% 95.5, 93.1 →
97.9, 95.3
LAPJV(test40) ⚪ NSD 1.99% 0.0181, 0.0181 →
0.0177, 0.0177
LAPJV(test400) ⚪ NSD 0.43% 3.23, 3.23 →
3.21, 3.22
MutualClusteringInfo(tr280) ⚪ NSD -0.56% 18.7, 19 →
18.6, 19.4
MutualClusteringInfo(tr50) ⚪ NSD -0.8% 17.4, 17.7 →
17.6, 18.1
NNIMaxStep(2000, normalize = TRUE) ⚪ NSD -0.01% 0.214, 0.213 →
0.214, 0.214
NNIMaxStep(300, normalize = TRUE) ⚪ NSD 0.06% 0.0316, 0.0316 →
0.0316, 0.0316
NNIMaxStep(300) ⚪ NSD -0.56% 0.00905, 0.00892 →
0.00904, 0.00901
PathDist(postTrees) ⚪ NSD 0.84% 2.31, 2.29 →
2.29, 2.26
PhylogeneticInfoDistance(tr180) ⚪ NSD -0.33% 58.1, 58.4 →
58.3, 58.5
PhylogeneticInfoDistance(tr50) 🟣 ~same -2.73% 43.1, 43.2 →
44.4, 44.3
RobinsonFoulds(tr280) ⚪ NSD 0.47% 4.77, 4.85 →
4.82, 4.76
RobinsonFoulds(tr280) ⚪ NSD 1.24% 4.62, 4.72 →
4.64, 4.58
RobinsonFoulds(tr50) ⚪ NSD -0.67% 4.58, 4.67 →
4.69, 4.61

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Memory Leak and Output Stalling when Processing Large Datasets with TreeDistance()

1 participant