feat: Add a per topic pending/processing limit - #790
Conversation
evanh
changed the title
There is now a config option that can be specified in each topic config, allowing each topic to have a maximum number of tasks in the processing/pending state. This allows a taskbroker that processes multiple topics to limit each topic individually, without impacting the throughput of other topics.
445fafa to
ded4a6c
Compare
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 76bc035. Configure here.
| if let Some(topic) = topic { | ||
| query_builder.push(" AND topic = "); | ||
| query_builder.push_bind(topic); | ||
| } |
There was a problem hiding this comment.
Invalid SQL when partitions are empty
High Severity
The topic filter is always appended as AND topic = ..., but add_partition_condition adds no WHERE when the assigned partition set is empty. That produces invalid SQL, and count_depths then panics via expect. Partition revoke runs before the writer flush, so a rebalance or shutdown that clears the last assignments hits this path whenever a per-topic limit is configured.
Reviewed by Cursor Bugbot for commit 76bc035. Configure here.
1 participant
There is now a config option that can be specified in each topic config, allowing each topic to have
a maximum number of tasks in the processing/pending state. This allows a taskbroker that processes multiple
topics to limit each topic individually, without impacting the throughput of other topics.