Skip to content

Rate limit the job submission request - #866

Open
hellolittlej wants to merge 1 commit into
masterfrom
rate-limit-job-submission
Open

Rate limit the job submission request#866
hellolittlej wants to merge 1 commit into
masterfrom
rate-limit-job-submission

Conversation

@hellolittlej

@hellolittlej hellolittlej commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

What

Adds an opt-in, per route based bucket rate limit.For jobsRoute, add the rate limit in front of the three v1 job-creation endpoints:

  • POST /api/v1/jobs
  • POST /api/v1/jobClusters/{cluster}/jobs
  • POST /api/v1/jobs/actions/quickSubmit

The rate

Configuration

Property Default Notes
mantis.master.api.v1.submitJob.throttle.enabled false Off by default; no limiter is constructed when off
mantis.master.api.v1.submitJob.permitsPerSecond 1000 Re-readable at runtime, no deploy needed
  • Guava RateLimiter, non-blocking tryAcquire(); token-bucket with the usual ~1s of burst accumulation, so a 1000/s limit admits up to a ~1000-request burst after an idle period.
  • Shed → HTTP 429, checked in BaseRoute.withThrottle:141 before the entity is unmarshalled — a storm costs a permit check, not a parse plus an actor ask.

Off by default because the right ceiling depends on cluster size and traffic —
operators opt in once they've measured their own submit rate. The rate is backed by a
LongDynamicProperty, so it re-reads on the dynamic-property refresh interval
(mantis.config.dynamic.refreshSecs, default 30s).

Metrics

MasterApiMetrics/throttledRequestCount gains a source tag (submitJob,
resourceCluster), so an operator can tell which limit is shedding. Previously the
route-level throttle and the resource-cluster gateway's RequestThrottledException
both incremented the same untagged counter.

Checklist

  • ./gradlew build compiles code correctly
  • Added new tests where applicable
  • ./gradlew test passes all tests
  • Extended README or added javadocs where applicable

@hellolittlej hellolittlej changed the title Rate limit the job submission request [TESTING] Rate limit the job submission request Jul 30, 2026
@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown

Test Results

814 tests  +17   803 ✅ +17   10m 24s ⏱️ +6s
167 suites + 3    11 💤 ± 0 
167 files   + 3     0 ❌ ± 0 

Results for commit 168d7d6. ± Comparison against base commit 25ff35c.

♻️ This comment has been updated with latest results.

@hellolittlej
hellolittlej force-pushed the rate-limit-job-submission branch 2 times, most recently from 9cfd8a5 to a8c0223 Compare August 4, 2026 20:30
@hellolittlej hellolittlej changed the title [TESTING] Rate limit the job submission request Rate limit the job submission request Aug 4, 2026
@hellolittlej
hellolittlej force-pushed the rate-limit-job-submission branch 3 times, most recently from ca6bbe8 to 8d0199a Compare August 5, 2026 06:43
@hellolittlej
hellolittlej force-pushed the rate-limit-job-submission branch from 8d0199a to 168d7d6 Compare August 11, 2026 18:06
jobRouteHandler,
system);
system,
ApiRequestRateLimiter.UNLIMITED);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can this be default so root signature remain the same?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is v0 route test but it's using the v1 job route with the legacy reason. ...

I prefer we not adding the overload just to accommodate a legacy test setup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants