Skip to content

fix: 0.0.23 release review (unique hash, nswitch threads, long-vector safety) - #106

Merged
davidbudzynski merged 5 commits into
fastverse:masterfrom
davidbudzynski:fix/0.0.23-release-review
Nov 2, 2026
Merged

davidbudzynski merged 5 commits into
fastverse:masterfrom
davidbudzynski:fix/0.0.23-release-review

Conversation

@davidbudzynski

Copy link
Copy Markdown
Collaborator

Summary

This is the 0.0.23 release-review follow-up. It fixes the issues found while checking the package against Seb's review comment: one real performance regression in the new growable duplicate tables, an nswitch threading race, and two long-vector edge cases. Codecov (#6/#67) is intentionally untouched.

CRAN currently has 0.0.21; 0.0.22 was never published, so this branch prepares 0.0.23.

Commits

  1. perf(unique): mix hash keys for sequential IDs — src/kit.h, tools/benchmark-unique.R
    The growable duplicate tables took the top bits of a multiplicative hash. Sequential integer IDs mapped to only about 16% of the slots in a power-of-two table, causing long linear-probe scans. Add a cheap xor/multiply mixer before indexing. Equality checks and output order are unchanged.

  2. fix(nswitch): make duplicate values deterministic with threads — src/nswitch.c + tests
    The loop over value/output pairs was parallel. With duplicate values, two pairs could write the same output position from different threads. The pair loop is now serial and atomic-output pairs are parallelised over elements instead. Also fixes list-input/logical-output indexing (i instead of j).

  3. fix(long vectors): guard logical fduplicated and class topn — src/dup.c, src/topn.c, docs + tests
    fduplicated() used an int loop index for logical vectors and could read out of bounds above INT_MAX elements; it now refuses those inputs with the standard long-vector error. topn() now applies xtfrm() before the long-vector heap path so classed long inputs follow order() semantics.

  4. docs(news): prepare 0.0.23 release notes — NEWS.md
    Version header and notes for the fixes above.

Benchmarks

Environment: macOS Sequoia 15.7, Apple silicon, R 4.6.2, single-threaded (kit.nThread = 1, OpenMP disabled for these runs), n = 10,000,000. Times are medians in seconds; 5 runs per case, 7 for the unique cases. “pre-fix” is 0.0.23 before this PR; “0.0.21” is the current CRAN release.

Inputs:

  • seq = 100,000 sequential integer IDs repeated to 10m (the case that triggered the regression).
  • random = 100,000 random integer IDs repeated to 10m.
  • iif / nif / switches as noted below.
Case 0.0.21 0.0.23 pre-fix this PR data.table
funique, sequential IDs 0.026 0.604 0.032 uniqueN 0.082
funique, random IDs 0.028 0.051 0.053 uniqueN 0.069
uniqLen, sequential IDs 0.024 0.595 0.032 —
fduplicated, sequential IDs 0.028 0.595 0.032 —
iif 0.016 0.016 0.015 fifelse 0.014
nif, 4 pairs 0.061 0.060 0.060 fcase 0.078
nswitch, 5 pairs 0.019 0.018 0.018 fcase 0.117
vswitch, character 0.130 0.120 0.124 —
vswitch, integer 0.018 0.017 0.017 —
topn(…, 6L) 0.010 0.004 0.004 —
psum, 3 vectors 0.013 0.011 0.011 —

The new hash restores the sequential-ID case from 0.604 s to 0.032 s, and is faster than data.table::uniqueN (0.082 s) for both sequential and random IDs. Sequential funique is now ~1.2x the old full-table version; random is ~1.9x. That remaining gap is the memory tradeoff from #74 (small growable tables instead of a table sized from the input length), and it is offset by the much lower peak memory.

nswitch thread scaling after the race fix (same 10m input, 5 pairs):

Threads Time
1 0.080 s
2 0.041 s
4 0.024 s

Reproduce the unique case with:

Rscript tools/benchmark-unique.R <library> <label> 10000000 100000

Correctness and checks

  • Full tinytest suite with OpenMP enabled (libomp): 1,625 assertions, 0 failures.
  • R CMD check --as-cran --no-manual on macOS / R 4.6.2: OK. The only warning is the local checkbashisms tool not being installed; running Debian's checkbashisms against configure and cleanup reports nothing.
  • New tests cover:
    • duplicate nswitch values with nThread = 1 vs 2,
    • list-input/logical-output indexing,
    • long logical fduplicated (opt-in long-vector test),
    • classed long topn error path (cheap ALTREP test).
  • Existing switch/psum/topn behaviour is unchanged in the benchmark table above.

The growable duplicate tables used the top bits of a multiplicative hash.
Sequential integer IDs mapped to only about 16% of the slots in a
power-of-two table, turning funique()/uniqLen()/fduplicated() into long
linear-probe scans.

Mix the key before indexing so the hash uses the table evenly. Equality
checks and output order are unchanged. Add tools/benchmark-unique.R to
keep the sequential-ID case visible.
Parallelise each atomic-output pair over the elements of x instead of
parallelising the loop over pairs. With duplicate values, two pairs
could previously write the same output slot from different threads, so
the last matching pair was scheduling-dependent. The last pair now wins
deterministically.

Also fix the list-input/logical-output path, which indexed the output
item by pair i instead of element j.
fduplicated() used an int loop index for logical vectors and stored
i + 1 in an int table slot, so a logical vector above INT_MAX elements
could read out of bounds. Refuse it with the standard long-vector error,
matching funique() and uniqLen().

Apply xtfrm() before topn()'s long-vector heap path so classed long
inputs follow order() semantics there too. Error clearly when xtfrm()
does not return an integer/double vector of the requested length, and
cover the cheap ALTREP error path in the long-vector tests.
@codecov-commenter

codecov-commenter commented Nov 2, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.84%. Comparing base (99413b6) to head (25a2a55).
⚠️ Report is 134 commits behind head on master.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #106      +/-   ##
==========================================
- Coverage   98.01%   96.84%   -1.17%     
==========================================
  Files          13       14       +1     
  Lines        5083     5203     +120     
==========================================
+ Hits         4982     5039      +57     
- Misses        101      164      +63     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The topn long-heap path scans more than INT_MAX elements, and the logical
fduplicated guards need an 8 GB vector. Both are covered only by the
opt-in KIT_TEST_LONG_VECTOR suite, so mark them # nocov like the other
long-vector guards.
@davidbudzynski
davidbudzynski merged commit 872f6c6 into fastverse:master Nov 2, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants