Repository navigation
fix(bigtable): synchronize MutationsBatcher staging and flush scheduling - #18827
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces thread safety to the MutationsBatcher class (both async and sync autogenerated versions) by using a reentrant lock (threading.RLock) to protect shared state, such as staged entries and flush metrics. It also ensures that staged entries are properly restored if a RuntimeError occurs during a flush scheduling attempt, and adds comprehensive unit tests to verify these thread-safety and error-recovery behaviors. There are no review comments to address, and we have no additional feedback to provide.
| should_flush = False | ||
| with self._stage_lock: | ||
| if self._closed.is_set(): | ||
| raise RuntimeError("Cannot append to closed MutationsBatcher") |
There was a problem hiding this comment.
We also have one of these at the start of the function. Are they both needed?
|
|
||
| Raises: | ||
| MutationsExceptionGroup: exception group with all unreported exceptions | ||
| """ |
There was a problem hiding this comment.
This also modifies some of the state. Does it need to use the lock?
a11b908 to
61d4792
Compare
…18807) Onboard google/cloud/observability/v1 API (google-cloud-observability). Internal tracking bug http://b/575514351 PiperOrigin-RevId: 1002511692 ## Debug Info ### System Details - Python version: Python 3.13.15 - OS: Linux 7.1.6-1rodete1-amd64 x86_64 - Librarian version: v0.49.0 ### Commands Executed - `python3 -m venv venv && source venv/bin/activate` - `pip install --upgrade pip` - `go run github.com/googleapis/librarian/cmd/librarian@v0.49.0 install` - `go run github.com/googleapis/librarian/cmd/librarian@v0.49.0 add google/cloud/observability/v1` - `go run github.com/googleapis/librarian/cmd/librarian@v0.49.0 generate -v google-cloud-observability` - `go run github.com/googleapis/librarian/cmd/librarian@v0.49.0 tidy` - `git add .` - `git commit -m "feat(google/cloud/observability/v1): add google-cloud-observability"`
61d4792 to
1831d04
Compare
Summary
Synchronizes mutation staging, flush scheduling, and shutdown in
MutationsBatcherusing a re-entrant lock (self._stage_lock = threading.RLock()):Prevents silent batch loss on
close()vs. timer flush race:_staged_entriesswap and task submission to_sync_flush_executorunder_stage_lock, and acquires_stage_lockinclose()when setting_closed, canceling_flush_timer, and scheduling the final flush before shutting down_sync_flush_executor._staged_entries,_staged_count, and_staged_bytesifcreate_taskraisesRuntimeError(e.g., if the executor was already shut down duringatexit) soclose()can flush them synchronously._stage_lock,_flush_jobs, and exception counters before starting the background_flush_timerdaemon thread.Prevents duplicate batch dispatch on concurrent
_schedule_flush()calls:append(),_timer_routine(),_schedule_flush(),close(), and_on_exit()via_stage_lockso user and background timer threads cannot read the same_staged_entriesreference and dispatch duplicateMutateRowsRPCs._entries_processed_since_last_raiseand_add_exceptionsupdates across concurrent flush worker threads and uses_flush_jobs.discardfor idempotent task cleanup.