Skip to content

feat(io): compile GooseFS metadata and page caches - #9263

Open
XuQianJin-Stars wants to merge 1 commit into
lance-format:mainfrom
XuQianJin-Stars:feat/goosefs-opendal-0.59.2-client-caches
Open

XuQianJin-Stars wants to merge 1 commit into
lance-format:mainfrom
XuQianJin-Stars:feat/goosefs-opendal-0.59.2-client-caches

Conversation

@XuQianJin-Stars

@XuQianJin-Stars XuQianJin-Stars commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Depend on opendal-service-goosefs 0.59.2 directly so Cargo unifies
metadata-cache and page-cache-io-uring into the same GooseFS client
build as opendal/services-goosefs.

Pin goosefs-sdk 0.2.2 in Lance (no OpenDAL bump). OpenDAL 0.59.2's
^0.2.1 requirement unifies onto 0.2.2. 0.2.2 keeps
metadata_cache_enabled off by default, matching Java
goosefs.user.metadata.cache.enabled, so compiling metadata-cache no
longer serves a 10-minute stale FileInfo for overwritten
_refs/tags/.json.

This compiles the caches into lance-io, pylance, and lance-jni whenever
the goosefs feature is enabled. Runtime behavior stays off until
configured:

  • Metadata cache: off by default. Set GOOSEFS_METADATA_CACHE_ENABLED=true to opt in
  • Page cache: off by default; enable with
    GOOSEFS_USER_CLIENT_CACHE_ENABLED=true and
    GOOSEFS_USER_CLIENT_CACHE_DIRS / goosefs.user.client.cache.dirs
  • page-cache-io-uring uses io_uring on Linux and tokio::fs elsewhere

Build with GooseFS (these also compile metadata-cache and page-cache-io-uring):

# Rust (goosefs is already in lance default features)
cargo build -p lance --features goosefs

# Python / pylance (python/Cargo.toml already enables lance/goosefs)
cd python && uv run make build

# Java JNI (dir-goosefs enables lance-io/goosefs)
cargo build --manifest-path java/lance-jni/Cargo.toml
# or: cd java && ./mvnw compile

Test plan

  • cargo test -p lance-io --features goosefs --lib object_store::providers::goosefs
  • cargo check --manifest-path python/Cargo.toml
  • cargo check --manifest-path java/lance-jni/Cargo.toml
  • Confirm goosefs-sdk in Cargo.lock is 0.2.2 and pulls lru / foyer / io-uring

@github-actions github-actions Bot added A-python Python bindings A-java Java bindings + JNI A-deps Dependency updates A-encoding Encoding, IO, file reader/writer enhancement New feature or request labels Sep 16, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-changes Latest Gatekeeper recommendation requests changes. and removed K-changes Latest Gatekeeper recommendation requests changes. labels Sep 16, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 16, 2026
@XuQianJin-Stars
XuQianJin-Stars force-pushed the feat/goosefs-opendal-0.59.2-client-caches branch from 0b8596f to 16c3067 Compare September 16, 2026 10:47
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 16, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 16, 2026
Depend on opendal-service-goosefs 0.59.2 so Cargo unifies
metadata-cache and page-cache-io-uring. Pin goosefs-sdk 0.2.2 so
metadata_cache_enabled stays off until GOOSEFS_METADATA_CACHE_ENABLED=true.
@XuQianJin-Stars
XuQianJin-Stars force-pushed the feat/goosefs-opendal-0.59.2-client-caches branch from 16c3067 to ee4cfe2 Compare September 16, 2026 11:01
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 16, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The rebased revision preserves the verified behavior: goosefs-sdk resolves to 0.2.2, both GooseFS caches are compiled, and metadata and page caching remain runtime opt-ins. Mutable tag freshness therefore remains protected by default.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-deps Dependency updates A-encoding Encoding, IO, file reader/writer A-java Java bindings + JNI A-python Python bindings enhancement New feature or request K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant