Repository navigation
feat(scheduler): add a setting to disable the shared file statistics cache - #2510
Merged
Merged
Conversation
…cache apache#2498 made the scheduler share one file statistics cache across sessions. Add ballista.scheduler.share_file_statistics_cache, true by default, so a session can keep its jobs off the shared cache if shared statistics turn out stale in a case the size and modification time check misses. A session with the setting off is returned as its session builder built it and never becomes the shared cache, so sessions left at the default keep sharing.
github-actions
Bot
added
the
documentation
milenkovicm
approved these changes
Sep 30, 2026
milenkovicm
left a comment
milenkovicm
left a comment
Contributor
There was a problem hiding this comment.
LGTM thanks @andygrove
7 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
No issue. This is the config follow-up to #2498 promised in review.
Rationale for this change
#2498 made the scheduler share one file statistics cache across sessions, so planning no longer reads every Parquet footer again for every job. A cached entry is only reused while the file's size and modification time match the job's own listing, and that check misses a file rewritten in place without changing either (apache/datafusion#25841 tracks comparing e-tags and versions). @milenkovicm asked for a
ballista.setting to turn sharing off should corner cases like that turn up.What changes are included in this PR?
ballista.scheduler.share_file_statistics_cache, true by default, and aBallistaConfig::share_file_statistics_cache()getter.share_file_statistics_cachereads the setting from the config each session is built with. A session with it set to false is returned as the session builder built it, which with the default builder means a statistics cache of its own, as before perf(scheduler): share the file statistics cache across sessions #2498. It never becomes the shared cache, so sessions left at the default keep sharing.share_file_statistics_cacheandBallistaCluster::new_memoryand the generated config reference describe the setting.A new test goes through
BallistaCluster::new_memoryand builds a session with the setting off, then two with the default. The two default sessions share a cache and the opted-out one keeps its own. Without the change it fails, because the opted-out session is built first and its cache becomes the shared one.Are there any user-facing changes?
There is a new setting. Running
SET ballista.scheduler.share_file_statistics_cache = false, or setting it on the client'sSessionConfig, keeps that session's jobs off the shared cache. It applies per session. Clients send everyballista.*setting with each job, so the client's value wins over any default set on the scheduler side, and there is no scheduler-wide switch.The default keeps the current behavior, and no existing API changes.