feat(algo): add GDS_PAGE_RANK backed by icebug (NetworKit, zero-copy Arrow)

[zachwinter]

Summary

First icebug (NetworKit) bridge for the algo extension: CALL GDS_PAGE_RANK('G') RETURN node, rank delegates PageRank to libnetworkit instead of the hand-rolled implementation. This is the start of wiring icebug into the CALL GDS_* paths we discussed on Discord (deprecating the hand-rolled algos in favor of icebug's zero-copy Arrow columnar engine).

Per that plan, the icebug-backed functions coexist with the existing algos under GDS_* names for one release cycle, then replace their internals in place. PAGE_RANK/PR are untouched.

How it works

projected graph → InMemGraph CSR (Louvain's fwd/bwd scan idiom)
  → arrow::UInt64Array → NetworKit::GraphR (zero-copy) → NetworKit::PageRank
  → scores streamed back through the existing GDSResultVertexCompute pipeline

The bind/register scaffolding is copied verbatim from page_rank.cpp (GDSFunction::initSharedState/getLogicalPlan/getPhysicalPlan); only the tableFunc body changes — it builds the CSR, hands it to icebug, and maps pr.scores() back to (node, rank) rows. Node offsets are dense in both worlds, so the mapping is identity.

Files

• algo/src/function/gds_page_rank.cpp — the bridge.
• algo/src/include/function/algo_function.h, algo/src/main/algo_extension.cpp — declare + register GDS_PAGE_RANK.
• algo/cmake/download_icebug.cmake — fetches the prebuilt icebug release (icebug-<os>-<arch> tag 12.8) into vendor/. Set -DICEBUG_SOURCE_DIR=<icebug checkout> to link a local source build instead (dev).
• algo/CMakeLists.txt, algo/src/function/CMakeLists.txt — link libnetworkit + Arrow::arrow_shared + OpenMP::OpenMP_CXX, with rpath to the vendored lib.
• algo/test/test_files/gds_page_rank.test.

Build requirements (needs your input)

The build now needs Apache Arrow C++ and OpenMP available (Arrow is a public dependency of libnetworkit). One thing to confirm for CI: the prebuilt icebug 12.8 links Arrow 24.x — the extension build needs a matching Arrow. Happy to adjust the vendoring (pin Arrow, or a newer icebug release built against the Arrow the CI uses) however you prefer. Locally I validated the full path with a source build of icebug against the system Arrow.

Testing

gds_page_rank.test (star graph): the hub ranks highest, and the ordering matches PAGE_RANK. Absolute values differ because NetworKit returns a sum-to-1 probability distribution (0.480 / 0.173…) vs the existing implementation's normalization — expected, and NetworKit's is the standard convention.

Notes / follow-ups

• Single-node-table graphs for now (multi-table is a small follow-up).
• GDS_LOUVAIN, GDS_NODE2VEC (ARRAY output → the vector extension's HNSW index), and personalized PageRank (icebug's built-in personalization/forSources) all follow this same pattern — one bridge, the whole NetworKit catalog.

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist

• I have added tests.
• I have requested a review from a maintainer.
• Documentation — deferred; happy to add once the surface/build approach is settled.

[adsharma]

Pin arrow works on other platforms, but not macosx. Bugscope has the same issue.

We'll likely have to update ladybug, icebug and all platform in sync unless we can figure out how to pin on homebrew.

[zachwinter]

I understand your comment in Discord now!

For now, I -SKIP'd gds_page_rank.test for the OFF build. Would you prefer a proper conditional (maybe a SKIP_ICEBUG_DISABLED token mirroring SKIP_COMPRESSION_DISABLED) so it runs automatically once icebug is enabled? I'll open a PR in lbug core if so.

[adsharma]

@zachwinter - icebug 13.0 out. Can you see if it fixes the problem here?

[zachwinter]

Tested 13.0 — short answer: it fixes macOS, but not this CI. And chasing it down turned up a separate crash that I think matters beyond this PR.

The version arithmetic

	links against
icebug 12.8	libarrow.so.2400 / libarrow.2400.dylib
icebug 13.0	libarrow.so.2500 / libarrow.2500.dylib
ladybug CI (pixi.toml)	libarrow = ">=23.0.1,<24"
Homebrew apache-arrow today	25.0.0_3

So the CI failure here is unchanged, just one major further away — the log is libarrow.so.2400: cannot open shared object file, and since it's the whole algo extension that then fails to dlopen, all ~45 algo tests go red, not only the GDS one.

macOS is genuinely better off, though: 12.8 wanted Arrow 24 after Homebrew had already moved past it, which is exactly the "can't pin on homebrew" problem you hit. 13.0 and Homebrew are both on 25 right now, so they line up.

What I verified on 13.0 (macOS arm64)

Built the extension against the prebuilt tarball (not a source build — the path CI would take), -DENABLE_ICEBUG=ON, system Arrow 25.0.0:

• The bridge compiles unchanged. Despite #42 removing the std::vector constructors, GraphR(count, bool, shared_ptr<UInt64Array>, shared_ptr<UInt64Array>) is byte-identical, and PageRank's new normalized / personalization args are defaulted.
• 71/71 algo tests pass, gds_page_rank included.

(Also noticed forSources / personalizationFromSources landed in 13.0 — that's the personalized-PR follow-up, I'll pick it up once this settles.)

The crash worth knowing about

While running the suite I hit a non-deterministic SIGSEGV — ~60% of runs, but only when GDS_PAGE_RANK runs after another algo test in the same process:

#1  _mi_thread_init + 391                     libarrow.2500.0.0.dylib
#3  __mi_malloc_generic + 363                 libarrow.2500.0.0.dylib
#5  arrow::MimallocAllocator::AllocateAligned libarrow.2500.0.0.dylib
...
#14 lbug::algo_extension::toU64              gds_page_rank.cpp:145

Arrow's default memory pool is mimalloc-backed, and mimalloc's per-thread bootstrap doesn't survive being reached from ladybug worker threads that already existed before libarrow came in via dlopen. Measured, same 3-test sequence:

• default pool: 5/8 runs crash
• ARROW_DEFAULT_MEMORY_POOL=system: 0/8
• passing arrow::system_memory_pool() to the builder in code: 0/10, full suite green

I observed this on macOS arm64; nothing about it looks macOS-specific, so I'd expect any dlopened extension that links Arrow to be exposed wherever the default pool is mimalloc. Worth flagging for Bugscope too if it links Arrow the same way.

The proposal: get Arrow off the ABI boundary entirely

Both problems — the version lockstep and the mimalloc crash — come from the extension having to link real libarrow just to hand icebug a CSR. Two ways out, in preference order:

1. Accept the Arrow C Data Interface at the GraphR entry point. A GraphR::fromCArrays(const ArrowArray*, const ArrowArray*, ...) taking the plain C structs. That interface is ABI-stable by design, the structs are header-only, and ladybug already ships them (src/common/arrow) — so the extension links zero Arrow, allocates through ladybug's own memory manager, and icebug's prebuilt can be built against whatever Arrow it likes. ladybug, icebug, Homebrew and conda-forge stop needing to move in sync at all.
2. Ship the prebuilt with Arrow statically linked + hidden visibility, so the tarball is self-contained. Fixes the version coupling, doesn't fix mimalloc, and puts a second Arrow in the process.

Happy to prototype (1) against icebug if you're open to it — it's a small surface, and it's the difference between "works today on 25" and "stops mattering".

Meanwhile, to get this PR green

I have the interim commit ready: an ENABLE_ICEBUG CMake option, default OFF, so a stock build links no icebug/Arrow/OpenMP and the extension loads everywhere — existing algos and their tests untouched — with GDS_* compiled and registered only when it's ON. I'll push that unless you'd rather wait and land the Arrow bump + icebug together.

On your Discord question about the test skip: I also have a SKIP_ICEBUG_DISABLED token written for ladybug core (mirrors SKIP_COMPRESSION_DISABLED), so gds_page_rank.test auto-skips when icebug is off and runs when it's on, instead of the blunt -SKIP I have now. Say the word and I'll open that core PR.