Add benchmarking and automatic core-count heuristics for af()/vis()/bglmnet()

[garthtarr]

Part of the parallelization improvement plan tracked in PARALLELIZATION.md (Priority 4, optional/advanced).

Split out of #16 so it can be scoped and closed independently of progress bars (#18) and backend selection (#19).

Background

Real benchmarking done while investigating #15 showed that for af() at n.c = 24, B = 30, actual future::multisession wall-clock time ran 250-370% over the naive ideal (total sequential work ÷ cores) at cores = 8/10, purely from worker-startup and per-future dispatch overhead — i.e. parallelizing was a net loss at that problem size. PARALLELIZATION.md's "Performance Considerations" section already flags this qualitatively (n < 200, p < 50 as a rough threshold) but it's currently just documentation; the functions themselves don't act on it.

Tasks

• Benchmark af(), vis(), and bglmnet() across a grid of problem sizes (n, p, B/n.c/nlambda) and core counts using bench::mark(), to establish where parallel overhead outweighs benefit for each function (their cost profiles likely differ)
• Based on the benchmarks, add an automatic heuristic (or at minimum a message()/warning()) that falls back to cores = 1 (or suggests it) when the estimated problem size is too small to benefit from multisession overhead
• Document the memory tradeoff: each worker holds its own copy of large objects (X, Xy, fitted models) — with N workers, expect roughly Nx memory footprint for large datasets. Add this to the relevant cores parameter documentation in af(), vis(), and bglmnet()
• Consider exposing the benchmarking script/results (e.g. as a vignette or inst/benchmarks/ script) so the heuristic thresholds are reproducible and can be re-validated as the package evolves

See PARALLELIZATION.md section "Performance Considerations" for existing (currently qualitative) guidance, and the af() benchmark discussion in #15 for methodology.

[garthtarr]

Benchmark findings so far (session notes, not yet acted on)

Ran af(), vis(), bglmnet() across a few problem-size regimes on a 10-core
machine to start scoping the heuristic. Results below; none of this has been
turned into code yet (no heuristic, no doc changes) - just benchmark evidence,
posting so it isn't lost.

Gotcha hit while benchmarking: an R console session had a stale, pre-#13/#15
build of mplot loaded in memory from earlier in the day (library(mplot) was
called before that session's later devtools::install() runs). This made
bglmnet() silently ignore cores entirely (it was still running the old
sequential-only code path) even though the installed package was correct.
Worth remembering when benchmarking interactively: reload (detach() +
library(), or a fresh session) after every reinstall, not just once at the
start.

Light workload (n in {100, 1000}, p = 6, B = 20 / n.c = 15 / nlambda = 30)

Speedup relative to cores = 1 (values < 1 mean parallel was slower):

fn	n	cores=2	cores=4	cores=8
af	100	0.96	1.11	0.95
af	1000	0.85	0.92	0.65
vis	100	0.45	0.44	0.34
vis	1000	0.48	0.43	0.30
bglmnet	100	0.15	0.14	0.10
bglmnet	1000	0.16	0.11	0.07

At "typical exploratory" sizes, parallelizing never won. bglmnet() was hit
hardest (down to 7-16% of sequential speed at cores=8) since each per-lambda
glmnet() fit is too cheap to amortize multisession dispatch/worker-startup
overhead. vis() lost 55-70%. af() was closest to break-even but still net
negative in 5 of 6 cells (the one >1 reading, af/n=100/cores=4, is likely just
noise from a single-rep timing).

Heavier workload (n = 2000, p = 10, B = 150 / n.c = 40 / nlambda = 100)

fn	cores=2	cores=4	cores=8
af	1.82	2.86	3.71
bglmnet	1.32	1.40	1.14
vis	0.93	0.84	0.82

Here af() shows real, scaling speedup - each worker's task (a full
B-replicate fence procedure) is substantial enough to amortize overhead.
bglmnet() gets a modest win at 2-4 cores but degrades again by 8. vis()
never won in either regime tested (bestglm/regsubsets per-replicate cost
appears to stay cheap even at n=2000, p=10).

Key implication for the heuristic: af(), vis(), and bglmnet() have
genuinely different parallelization profiles, not just different numeric
thresholds along the same one-dimensional "problem size" dial. A single
shared heuristic/threshold across all three is likely wrong; vis() in
particular may need its own logic (or a much more aggressive default) since
no configuration tested so far showed it benefiting from cores > 1.

p = 20 (n = 2000, B = 150 / n.c = 40 / nlambda = 100) - af() blowup

cores	af()	vis()	bglmnet()
1	152s	14.6s	9.2s
2	87s (1.75x faster)	15.3s	5.4s
4	2042s (13x slower than cores=1)	20.7s	7.5s
8	4770s (31x slower than cores=1, ~80 min)	18.8s	5.8s

This is a much bigger concern than the light-workload overhead story: af()
goes from a genuine win at cores=2 to a catastrophic, non-monotonic blowup at
4-8 cores. Not yet root-caused - best guess (unverified, didn't want to spend
more wall-clock confirming it) is memory/resource contention: at p=20,
af()'s default nvmax is kf = 21, so each of the n.c workers considers a
much larger model space per bootstrap replicate and accumulates growing
fence.mod lists on top of its own copy of Xy/mfstar; running 4-8 such
workers concurrently could exceed available memory and trigger OS-level
swapping, which would produce exactly this kind of cliff rather than a
graceful slowdown. vis()/bglmnet() didn't show this, plausibly because
their workers are much lighter (a single regsubsets/glmnet call per task,
no accumulating model lists).

This needs to be treated as a distinct failure mode from the light-workload
overhead issue - not just "small problems shouldn't parallelize" but
possibly "af() with large p/nvmax + high core count can catastrophically
regress," which is arguably more dangerous since it can silently turn a
90-second job into a 90-minute one. Should be root-caused (e.g. monitor RSS
per worker) before deciding on a mitigation, rather than guessing at a p-based
cap.

Open questions for next session

• Heuristic behavior for the (more common) light-workload case: message-only
  suggesting cores = 1 (never override the user's explicit argument),
  silent auto-fallback only when cores is left at its default, or a louder
  warning()?
• Should vis() get a more aggressive default (e.g. always suggest
  cores = 1) given no tested configuration showed it benefiting?
• The p=20 af() blowup needs root-causing before any mitigation is designed -
  candidate next step is monitoring per-worker memory (RSS) across cores to
  confirm/rule out swapping as the cause.