Explore a real multi-Python-version build matrix (osx-arm64)

[Tobias-Fischer]

Summary

Follow-up on RoboStack/robostack.github.io#100. Depends on RoboStack/vinca#153, which stops vinca from adding a Python host/run dependency to every recipe unconditionally — only packages that actually need Python (ament_python, a rosidl interface package, or an explicit Python-flavored dependency) get one now. That's what makes a real multi-version python: pin viable at all: without it, every single recipe (including pure C++ libraries with zero Python content — the majority, per @traversaro's investigation in the linked issue) would get rebuilt once per Python version.

This PR is scoped as an experiment/proposal, not a request to flip the pin for the full distro — see "Open questions" below.

What changed

• vinca_pinning.yaml/conda_build_config.yaml: python: changed from a single pinned version to a real 3-entry matrix (3.11, 3.12, 3.13). Note the build-string glob differs by version — conda-forge only started tagging the build string with the minor version (*_cp313, to disambiguate from the free-threaded *_cp313t variant) starting with 3.13; 3.11/3.12 still publish under the older, untagged *cpython suffix.
• pixi.toml: added sccache as a dependency and --no-build-id to the build task (required for sccache/ccache to actually get cache hits, since both are sensitive to the timestamped build-directory paths rattler-build uses by default), added a sccache-stats task, and pointed the vinca pypi-dependency at RoboStack/vinca#153's branch pending its own review.
• README-multi-python.md: full writeup of the motivation, changes, validation results, and open questions.

Validation

Rather than trying to build the full ~900-package distro, this was validated against a representative 133-package subset (std_msgs, example_interfaces, rclcpp, rclpy, demo_nodes_cpp, demo_nodes_py, launch, ros2cli, ros2topic, plus their transitive closure — deliberately spanning all three of vinca's Python-need categories), osx-arm64 only, built locally end-to-end:

• 77 packages built exactly once regardless of the 3-version matrix (rclcpp, rmw_*, rcutils, cyclonedds, iceoryx*, fastcdr, demo_nodes_cpp, gtest/gmock-vendor, rosidl_default_generators, ...).

• 56 packages built 3x, once per Python version (all *_msgs packages, rclpy, launch*, ros2cli, ros2topic, demo_nodes_py, the ament_* lint tools, rosidl_adapter/cli/parser/generator_*, pybind11_vendor, and — as an accepted false positive — fastrtps).

• 245 total build artifacts (77×1 + 56×3 = 245), exactly matching the classification with zero exceptions.

• sccache measurably reuses compiled objects across Python-version rebuilds of the same package. Real wall-clock timings, isolated single-recipe rebuilds (same build command each time, no --skip-existing so all 3 Python variants are genuinely rebuilt):

  std_msgs — a small message package, mostly CMake configure/rosidl codegen rather than compilation:

  Scenario	py3.11	py3.12	py3.13
  No sccache (fresh each time)	55s	48s	37s
  With sccache (cold → warm within one run)	47s	35s	28s

  fastrtps — a large, compile-heavy vendored C++ library (also Python-classified, an accepted false positive), same methodology:

  Scenario	py3.11	py3.12	py3.13
  No sccache (fresh each time)	57s	58s	61s
  With sccache (cold → warm within one run)	66s	22s	22s

  confirmed via sccache --show-stats: 645 compile requests, 430 cache hits (66.67%). The first variant pays a small cache-write penalty (66s vs. 57s cold), but the second and third — which reuse most of the compiled objects across the Python-version-specific bindings layer — are ~2.6-2.8× faster than a fresh, uncached rebuild. std_msgs alone understated the benefit: most of its build time is CMake configure/codegen, not compilation, so the payoff scales with how much actual C/C++ compilation a package does — much more pronounced for something like fastrtps than for a small message package.

Full details, including the exact per-category package lists, are in README-multi-python.md.

Open questions for reviewers

• CI cost: flipping the entire distro's python: pin to this 3-version matrix would 3x the build time for whatever fraction of the ~900 packages the heuristic classifies as Python-dependent (roughly 40% on this subset, though the full-distro fraction hasn't been measured). This seems like a decision worth making deliberately rather than as a side effect of merging vinca#153 — that PR alone is harmless with a single-version pin (it just stops adding a spurious Python dependency to non-Python packages, freeing up e.g. Windows/Linux platforms from unnecessary rebuild churn even without touching this pin).
• ament_cmake_test-shaped edge cases: any package that calls ament_python_install_package() internally without declaring a Python dependency in package.xml builds its Python helper once, at whichever Python version happens to be python_min — a downstream package importing that helper while running under a different active Python version could see a site-packages path mismatch. Not exercised in this validation run (--test skip throughout); worth flagging for anyone relying on it in production. See vinca#153's discussion for the $SP_DIR fallback this surfaced.
• Is 3 versions (3.11/3.12/3.13) the right starting matrix, or should it track conda-forge's currently-supported set more closely (e.g. include 3.14)?

Test plan

• Full local build of the 133-package subset described above, osx-arm64, 3-version Python matrix — verified 245/245 expected artifacts, correct 1x/3x split with zero exceptions
• Feedback from maintainers on whether/how to extend this to more of the distro, and on the open questions above

🤖 Generated with Claude Code

[Tobias-Fischer]

Force-pushed a corrected history: the original push accidentally included ~110 unrelated commits (my local branch was based off a stale, unmerged integration branch rather than main). Rebased cleanly onto current main and re-validated -- this also surfaced one more real fix, now in vinca (Tobias-Fischer/vinca@2cfe132): main's conda_build_config.yaml is a minimal hand-maintained file that doesn't define python_min at all, unlike the fuller pinning setup this was originally tested against. Made vinca's build-time Python entry default to 3.11 when python_min is undefined instead of producing an invalid match spec. PR now shows a clean 4-file diff.

[Tobias-Fischer]

CI caught a real, separate issue: main's currently-committed rosdistro_snapshot.yaml predates vinca's snapshot-metadata-caching feature (RoboStack/vinca#134/#145) and lacks per-package dependency metadata that newer vinca commits require -- since this PR bumps vinca to a fork branch off current master, recipe generation broke outright for every package ("Snapshot metadata for 'X' has no dependencies"). Regenerated it (pixi run create_snapshot) and verified full recipe generation now succeeds.

[Tobias-Fischer]

Two distinct failures in the last run, both confirmed pre-existing on main and unrelated to this PR's actual changes:

1. linux-64/win-64/osx-64/linux-aarch64: dependency resolution fails because ros-humble-rclcpp (needed transitively by moveit_py, which main's vinca.yaml already selects) can't be installed -- the published robostack-staging builds require numpy <2.0, but main's conda_build_config.yaml already pins numpy: 2 unconditionally. Verified via git show origin/main that both the numpy: 2 pin and the moveit_py selection predate this PR entirely -- I never touched either. This looks like the kind of staleness Full rebuild September 2026 + Sync cross-distribution Vinca package coverage #420 ("Full rebuild September 2026 + Sync cross-distribution Vinca package coverage") is meant to address; not something I can fix from this PR without rebuilding a large chunk of the published channel.
2. win-64 only: fixed (488c411) -- a UnicodeEncodeError in vinca's own generation-summary printing (Windows' console defaults to cp1252, can't encode the Unicode ellipsis rich uses for truncated table cells). pixi.toml was missing PYTHONIOENCODING=utf-8.

Given (1), I don't expect a clean full-distro CI run here regardless of anything in this PR -- the local 133-package validation (README-multi-python.md) used freshly-built local artifacts throughout, so it never exercised this particular staleness. Flagging clearly rather than chasing it further, since resolving it would mean rebuilding a chunk of the published distro unrelated to the Python-matrix proposal this PR is actually about.

[Tobias-Fischer]

Addressed a review comment on the companion vinca PR (RoboStack/vinca#153): @traversaro correctly flagged that auto-detecting sccache on PATH silently changes build behavior depending on the environment. Fixed there (1bc88d1) -- sccache routing now requires an explicit VINCA_USE_SCCACHE=1. This branch (569fc1e) updated to match: pixi.toml's build task now sets that env var explicitly rather than relying on sccache merely being installed. Verified end-to-end with a freshly-reset sccache server: no env var → zero compile requests despite sccache being installed; env var set → 100% cache hit rate on a repeat build.

Also tried a second package to strengthen the sccache demonstration: rclpy (genuinely Python-dependent with real compiled pybind11 bindings, unlike fastrtps which is an accepted false-positive for needing Python). Couldn't get a clean isolated build locally -- its dependency chain needs rmw_cyclonedds_cpp, whose cyclonedds build requires openssl >=4.0.2, not resolvable in the currently-configured channels (only 3.x available). Reproducible from a fully clean rebuild, so a genuine pre-existing gap in the conda-forge/robostack channel state, unrelated to this PR -- same category as the numpy/rclcpp staleness already flagged above. fastrtps remains the demonstrated example in README-multi-python.md.