Skip to content

fix: AR clear statement cache after failover - #194

Merged
aaron-congo merged 3 commits into
mainfrom
fix/ar-clear-statement-cache-after-failover
Oct 5, 2026
Merged

aaron-congo merged 3 commits into
mainfrom
fix/ar-clear-statement-cache-after-failover

Conversation

@aaron-congo

Copy link
Copy Markdown
Contributor

Summary

fix: clear ActiveRecord's prepared statement cache after failover

Description

In 1.0.0, after a successful failover, every query ActiveRecord had already prepared fails on that connection with AwsAdvancedRubyDriverWrapper::Errors::AwsError: Method invoked against old connection. It keeps failing for as long as the connection lives. Queries not prepared before the failover still work.

This affects:

  • aws_postgresql, which has prepared statements on by default. In practice, every pooled connection that survives a failover becomes unusable for its existing queries until the process restarts.
  • aws_mysql2 with prepared_statements: true, which the KMS encryption docs tell MySQL users to enable.

The stock postgresql and mysql2 adapters recover from the same connection loss without errors.

Root cause:

  1. After a successful failover, the adapters reuse the same connection object, which the wrapper has reconnected to a new physical connection, and re-run configure_connection. They did not clear ActiveRecord's prepared statement cache.
  2. ActiveRecord therefore kept executing statements by names (e.g. a1) that only existed on the old physical connection, and the wrapper's bounded-connection check rejected each one.
  3. That error is a generic AwsError, which translate_exception returns unchanged, and the connection still reports active?, so the pool keeps handing it out.

Fix: for FailoverSuccessError and TransactionStateUnknownError, translate_exception now calls clear_cache!(new_connection: true) before configure_connection. This is ActiveRecord's own hook for a connection that has been replaced: it forgets the cached statements without trying to deallocate them on the new connection, so they are prepared again on first use. It exists with the same signature in ActiveRecord 7.2, 8.0, and 8.1.

Tests:

  • New unit examples in both adapter specs check that the cache is cleared before the connection is reconfigured, for both errors. They failed before the change.
  • New integration example in failover_activerecord_spec.rb: a prepared find (with prepared_statements: true) before and after a writer crash, expecting FailoverSuccessError and then successful repeats of the same query. The existing failover tests only ran plain SQL through select_value, which never uses prepared statements, so they couldn't catch this.
  • Unit suite passes on the ActiveRecord 7.2, 8.0, and 8.1 gemfiles; RuboCop is clean.
  • Manual check against Aurora PostgreSQL 17 and Aurora MySQL 8.4 (3 instances each)
    • Before the fix, the same prepared find failed on every attempt after the FailoverSuccessError, on both engines.
    • After the fix, it succeeds on every attempt.
    • The stock adapters recovered in both cases.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

After a successful failover the ActiveRecord adapters reuse the same connection object, which the wrapper has reconnected to a new physical connection, and re-run configure_connection. They did not clear ActiveRecord's prepared statement cache, so ActiveRecord kept executing statements by names that only existed on the old physical connection. The wrapper rejected each one with "Method invoked against old connection", and because the connection still looked healthy, every query prepared before the failover failed on it until the process restarted.

translate_exception now calls clear_cache!(new_connection: true) before configure_connection for FailoverSuccessError and TransactionStateUnknownError, so those statements are prepared again on the new connection.
…t-cache-after-failover

# Conflicts:
#	CHANGELOG.md
@aaron-congo
aaron-congo requested a review from a team as a code owner October 5, 2026 23:15
@sophia-bq sophia-bq changed the title Fix/ar clear statement cache after failover fix: AR clear statement cache after failover Oct 5, 2026
@aaron-congo
aaron-congo merged commit fe49398 into main Oct 5, 2026
10 checks passed
@aaron-congo
aaron-congo deleted the fix/ar-clear-statement-cache-after-failover branch October 5, 2026 23:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants