Skip to content

fix(bigtable): synchronize MutationsBatcher staging and flush scheduling - #18827

Merged
mutianf merged 4 commits into
mainfrom
fix-batcher-sync-flush-races
Oct 23, 2026
Merged

mutianf merged 4 commits into
mainfrom
fix-batcher-sync-flush-races

Conversation

@mutianf

@mutianf mutianf commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Synchronizes mutation staging, flush scheduling, and shutdown in MutationsBatcher using a re-entrant lock (self._stage_lock = threading.RLock()):

  1. Prevents silent batch loss on close() vs. timer flush race:

    • Protects _staged_entries swap and task submission to _sync_flush_executor under _stage_lock, and acquires _stage_lock in close() when setting _closed, canceling _flush_timer, and scheduling the final flush before shutting down _sync_flush_executor.
    • Restores _staged_entries, _staged_count, and _staged_bytes if create_task raises RuntimeError (e.g., if the executor was already shut down during atexit) so close() can flush them synchronously.
    • Initializes _stage_lock, _flush_jobs, and exception counters before starting the background _flush_timer daemon thread.
  2. Prevents duplicate batch dispatch on concurrent _schedule_flush() calls:

    • Serializes append(), _timer_routine(), _schedule_flush(), close(), and _on_exit() via _stage_lock so user and background timer threads cannot read the same _staged_entries reference and dispatch duplicate MutateRows RPCs.
    • Protects _entries_processed_since_last_raise and _add_exceptions updates across concurrent flush worker threads and uses _flush_jobs.discard for idempotent task cleanup.

@mutianf
mutianf requested a review from a team as a code owner October 23, 2026 14:59
@mutianf
mutianf added this pull request to stack #18828 October 23, 2026 14:59

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces thread safety to the MutationsBatcher class (both async and sync autogenerated versions) by using a reentrant lock (threading.RLock) to protect shared state, such as staged entries and flush metrics. It also ensures that staged entries are properly restored if a RuntimeError occurs during a flush scheduling attempt, and adds comprehensive unit tests to verify these thread-safety and error-recovery behaviors. There are no review comments to address, and we have no additional feedback to provide.

should_flush = False
with self._stage_lock:
if self._closed.is_set():
raise RuntimeError("Cannot append to closed MutationsBatcher")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We also have one of these at the start of the function. Are they both needed?


Raises:
MutationsExceptionGroup: exception group with all unreported exceptions
"""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also modifies some of the state. Does it need to use the lock?

Base automatically changed from docs-batcher-flush-exceptions to main October 23, 2026 17:24
@mutianf
mutianf force-pushed the fix-batcher-sync-flush-races branch 2 times, most recently from a11b908 to 61d4792 Compare October 23, 2026 18:32
suztomo and others added 4 commits October 23, 2026 14:42
…18807)

Onboard google/cloud/observability/v1 API (google-cloud-observability).

Internal tracking bug http://b/575514351

PiperOrigin-RevId: 1002511692

## Debug Info
### System Details
- Python version: Python 3.13.15
- OS: Linux 7.1.6-1rodete1-amd64 x86_64
- Librarian version: v0.49.0

### Commands Executed
- `python3 -m venv venv && source venv/bin/activate`
- `pip install --upgrade pip`
- `go run github.com/googleapis/librarian/cmd/librarian@v0.49.0 install`
- `go run github.com/googleapis/librarian/cmd/librarian@v0.49.0 add
google/cloud/observability/v1`
- `go run github.com/googleapis/librarian/cmd/librarian@v0.49.0 generate
-v google-cloud-observability`
- `go run github.com/googleapis/librarian/cmd/librarian@v0.49.0 tidy`
- `git add .`
- `git commit -m "feat(google/cloud/observability/v1): add
google-cloud-observability"`
@mutianf
mutianf force-pushed the fix-batcher-sync-flush-races branch from 61d4792 to 1831d04 Compare October 23, 2026 18:42
@mutianf
mutianf enabled stacked auto-merge (squash) October 23, 2026 18:43
@mutianf
mutianf merged commit 38afa8a into main Oct 23, 2026
57 checks passed
@mutianf
mutianf deleted the fix-batcher-sync-flush-races branch October 23, 2026 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants