Clamp Pareto-smoothed Pathfinder init weights at 0

[avehtari]

Fixes #1323.

This PR was assisted by Claude

Summary

$sample(init = pathfinder_fit) can fail with Assertion on 'weights' failed: Element <i> is not >= 0. This PR sets negative Pareto-smoothed weights in process_init_approx() to 0 before posterior::resample_draws().

Cause

process_init_approx() Pareto-smooths the importance weights on the natural scale, exp(lp__ - lp_approx__ - max(...)). When most Pathfinder paths are poor, most of these values underflow to exactly 0. With posterior 1.7.1 and earlier, pareto_smooth() can then return weights of about -2e-16, and resample_draws() rejects them. stan-dev/posterior#468 fixes the cause in posterior. This PR protects users with a posterior version without that fix.

Change

In process_init_approx() (R/args.R), after the call to posterior::pareto_smooth():

weights <- pmax(weights, 0)

The change applies only to the branch that Pareto-smooths the weights. The values that the clamp changes are of the order of 1e-16, so the selection probabilities do not change in practice. The rowsum() collapse of duplicate draws that follows then sums non-negative values.

Tests

New test in tests/testthat/test-fit-init.R, Pathfinder init weights are not negative after Pareto smoothing. It uses a fake Pathfinder fit, as the existing tests in that file do, so it does not need CmdStan.

• 400 draws with 350 log weights below -1000 and 50 log weights between 0 and -60, with the real posterior::pareto_smooth() and resample_draws(). With posterior 1.7.1 on master, this gives the error Assertion on 'weights' failed: Element 341 is not >= 0.
• The same draws with a mocked pareto_smooth() that returns negative weights, and with three duplicate draws. This part tests the clamp with any posterior version, also with a version that contains Remove the offset at a tied Pareto tail cutoff posterior#468.

On master, the new test gives 3 failures. With this PR, it passes.

Tested versions

• cmdstanr: this branch, based on upstream master 55342f4f.
• posterior 1.7.1 (CRAN), R 4.6.1, testthat 3.3.2, Linux x86_64.
• With NOT_CRAN=true, the new test and the existing tests in test-fit-init.R that use fake fits pass (Pathfinder fit initializations use the intended weights, Pathfinder init candidates are distinct target parameter vectors, VB and Laplace inits error with the right algorithm label, draws inits are recycled in order when too few are supplied), and their snapshots do not change. I did not run the tests that compile and fit models locally.

Copyright and Licensing

Please list the copyright holder for the work you are submitting
(this will be you or your assignee, such as a university or company):
Aalto University

By submitting this pull request, the copyright holder is agreeing to
license the submitted work under the following licenses:

• Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
• Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

[jgabry]

I changed this to target the v1.0 branch (instead of master) so I could see if there are any merge conflicts with v1.0. Looks like there is a conflict. I'll fix the merge conflict and push.

[jgabry]

I changed this to target the v1.0 branch, fixed the merge conflict, and simplified the test a little bit. I also added a comment about removing this once we require a version of posterior with the fix. When CI passes we can merge this into the v1.0 branch.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (v1.0@3cfa89a). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             v1.0    #1324   +/-   ##
=======================================
  Coverage        ?   93.95%           
=======================================
  Files           ?       19           
  Lines           ?     7656           
  Branches        ?        0           
=======================================
  Hits            ?     7193           
  Misses          ?      463           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.