Repository navigation
fix: 0.0.23 release review (unique hash, nswitch threads, long-vector safety) - #106
Merged
davidbudzynski merged 5 commits intoNov 2, 2026
Merged
Conversation
The growable duplicate tables used the top bits of a multiplicative hash. Sequential integer IDs mapped to only about 16% of the slots in a power-of-two table, turning funique()/uniqLen()/fduplicated() into long linear-probe scans. Mix the key before indexing so the hash uses the table evenly. Equality checks and output order are unchanged. Add tools/benchmark-unique.R to keep the sequential-ID case visible.
Parallelise each atomic-output pair over the elements of x instead of parallelising the loop over pairs. With duplicate values, two pairs could previously write the same output slot from different threads, so the last matching pair was scheduling-dependent. The last pair now wins deterministically. Also fix the list-input/logical-output path, which indexed the output item by pair i instead of element j.
fduplicated() used an int loop index for logical vectors and stored i + 1 in an int table slot, so a logical vector above INT_MAX elements could read out of bounds. Refuse it with the standard long-vector error, matching funique() and uniqLen(). Apply xtfrm() before topn()'s long-vector heap path so classed long inputs follow order() semantics there too. Error clearly when xtfrm() does not return an integer/double vector of the requested length, and cover the cheap ALTREP error path in the long-vector tests.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #106 +/- ##
==========================================
- Coverage 98.01% 96.84% -1.17%
==========================================
Files 13 14 +1
Lines 5083 5203 +120
==========================================
+ Hits 4982 5039 +57
- Misses 101 164 +63 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The topn long-heap path scans more than INT_MAX elements, and the logical fduplicated guards need an 8 GB vector. Both are covered only by the opt-in KIT_TEST_LONG_VECTOR suite, so mark them # nocov like the other long-vector guards.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is the 0.0.23 release-review follow-up. It fixes the issues found while checking the package against Seb's review comment: one real performance regression in the new growable duplicate tables, an
nswitchthreading race, and two long-vector edge cases. Codecov (#6/#67) is intentionally untouched.CRAN currently has 0.0.21; 0.0.22 was never published, so this branch prepares 0.0.23.
Commits
perf(unique): mix hash keys for sequential IDs—src/kit.h,tools/benchmark-unique.RThe growable duplicate tables took the top bits of a multiplicative hash. Sequential integer IDs mapped to only about 16% of the slots in a power-of-two table, causing long linear-probe scans. Add a cheap xor/multiply mixer before indexing. Equality checks and output order are unchanged.
fix(nswitch): make duplicate values deterministic with threads—src/nswitch.c+ testsThe loop over value/output pairs was parallel. With duplicate values, two pairs could write the same output position from different threads. The pair loop is now serial and atomic-output pairs are parallelised over elements instead. Also fixes list-input/logical-output indexing (
iinstead ofj).fix(long vectors): guard logical fduplicated and class topn—src/dup.c,src/topn.c, docs + testsfduplicated()used anintloop index for logical vectors and could read out of bounds aboveINT_MAXelements; it now refuses those inputs with the standard long-vector error.topn()now appliesxtfrm()before the long-vector heap path so classed long inputs followorder()semantics.docs(news): prepare 0.0.23 release notes—NEWS.mdVersion header and notes for the fixes above.
Benchmarks
Environment: macOS Sequoia 15.7, Apple silicon, R 4.6.2, single-threaded (
kit.nThread = 1, OpenMP disabled for these runs),n = 10,000,000. Times are medians in seconds; 5 runs per case, 7 for the unique cases. “pre-fix” is 0.0.23 before this PR; “0.0.21” is the current CRAN release.Inputs:
seq= 100,000 sequential integer IDs repeated to 10m (the case that triggered the regression).random= 100,000 random integer IDs repeated to 10m.iif/nif/ switches as noted below.funique, sequential IDsuniqueN0.082funique, random IDsuniqueN0.069uniqLen, sequential IDsfduplicated, sequential IDsiiffifelse0.014nif, 4 pairsfcase0.078nswitch, 5 pairsfcase0.117vswitch, charactervswitch, integertopn(…, 6L)psum, 3 vectorsThe new hash restores the sequential-ID case from 0.604 s to 0.032 s, and is faster than
data.table::uniqueN(0.082 s) for both sequential and random IDs. Sequentialfuniqueis now ~1.2x the old full-table version; random is ~1.9x. That remaining gap is the memory tradeoff from #74 (small growable tables instead of a table sized from the input length), and it is offset by the much lower peak memory.nswitchthread scaling after the race fix (same 10m input, 5 pairs):Reproduce the unique case with:
Correctness and checks
R CMD check --as-cran --no-manualon macOS / R 4.6.2: OK. The only warning is the localcheckbashismstool not being installed; running Debian'scheckbashismsagainstconfigureandcleanupreports nothing.nswitchvalues withnThread = 1vs2,fduplicated(opt-in long-vector test),topnerror path (cheap ALTREP test).psum/topnbehaviour is unchanged in the benchmark table above.