feat(spanner): add retry and resume support for BatchWrite

[olavloite]

Add configurable retry and backoff policies to BatchWriteTransactionBuilder and implement client-side stream resumption in BatchWriteResponseStream.

Because Spanner's BatchWrite RPC does not include built-in resume tokens, the client now tracks unacknowledged mutation groups and automatically re-establishes streams on transient errors:

• Resends only remaining unprocessed mutation groups on retry attempts.
• Maps relative response indices on retry streams back to original input indices.
• Releases mutation group payload memory immediately upon acknowledgment.
• Rotates channel hints on retry to avoid pinning to unhealthy subchannels.

[gemini-code-assist]

Code Review

This pull request enhances the Spanner BatchWriteTransaction by implementing automatic retries and backoff policies during streaming execution. It introduces configuration options for attempt timeouts, retry policies, and backoff policies, and refactors BatchWriteResponseStream to track completed mutation groups and handle stream reconnection. Feedback on the changes suggests that a premature stream EOF should be treated as a retryable transient error (e.g., Code::Unavailable) rather than immediately returning a terminal error, allowing the client to re-establish the stream and resume processing.

[codecov]

Codecov Report

❌ Patch coverage is 99.69849% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.43%. Comparing base (8535323) to head (c317204).

Files with missing lines	Patch %	Lines
src/spanner/src/batch_write_transaction.rs	99.69%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6513      +/-   ##
==========================================
+ Coverage   96.39%   96.43%   +0.04%     
==========================================
  Files         301      301              
  Lines       84762    85689     +927     
==========================================
+ Hits        81706    82638     +932     
+ Misses       3056     3051       -5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[olavloite]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds automatic retry and backoff capabilities to the BatchWriteTransaction::execute_streaming method by refactoring BatchWriteResponseStream into a stateful stream that tracks completed mutation groups and translates relative indices. It also exposes builder methods to configure timeouts, retry, and backoff policies, backed by comprehensive unit tests. The feedback highlights a bug in metrics tracking where recording a successful attempt on the first received message prematurely prevents subsequent midway stream failures from being captured.

[olavloite]

/gemini review

[gemini-code-assist]

Code Review

This pull request refactors the BatchWriteTransaction and BatchWriteResponseStream in the Spanner client to support automatic retries and backoff policies for streaming batch write operations. It introduces GaxRequestOptions to configure attempt timeouts, retry policies, and backoff policies, and implements a robust state machine in BatchWriteResponseStream to handle transient stream failures, translate relative indices back to original positions, and record observability metrics. Additionally, a comprehensive suite of unit tests has been added to verify various success, failure, and retry scenarios. I have no feedback to provide.

[olavloite]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds automatic retries, backoff policies, and observability metrics to the Spanner BatchWrite streaming transaction. It updates BatchWriteTransactionBuilder and BatchWriteTransaction to accept custom GAX request options and implements a robust BatchWriteResponseStream that tracks completed mutation groups, handles stream reconnections, and records metrics. Comprehensive unit and mock tests are also added. Feedback is provided to clear the headers field at the start of each stream establishment attempt to avoid reporting stale headers in observability metrics if a subsequent connection attempt fails.

[olavloite]

/gemini review

[gemini-code-assist]

Code Review

This pull request refactors the BatchWrite transaction streaming implementation in the Spanner client to support automatic retries and backoff policies for transient network or server errors. It introduces GaxRequestOptions configuration to BatchWriteTransactionBuilder and BatchWriteTransaction, allowing users to customize attempt timeouts, retry policies, and backoff policies. The BatchWriteResponseStream has been significantly enhanced to track completed mutation groups, translate relative response indices back to the caller's original indices, record observability metrics (operation and attempt counts/latencies), and handle stream cancellation on drop. Additionally, comprehensive unit and mock tests have been added to verify the retry logic, error handling, and metrics recording. I have no feedback to provide as there are no review comments to assess.