Skip to content

fix: recover redis connections and preserve encrypted names - #58

Open
lorenzocorallo wants to merge 1 commit into
mainfrom
fix/incident-20260917-data-stability
Open

lorenzocorallo wants to merge 1 commit into
mainfrom
fix/incident-20260917-data-stability

Conversation

@lorenzocorallo

@lorenzocorallo lorenzocorallo commented Sep 17, 2026 •

Copy link
Copy Markdown
Member

Redis outages permanently closed the backend client after its retry budget expired. Keep reconnecting with capped exponential backoff and jitter. Valid Unicode Telegram names also exceeded the encrypted-name columns: a 64-character Chinese name becomes 442 ciphertext characters. Widen both name columns from 192 to 512 and record migration 0019 without truncating existing values.

Validation: 24 tests, type checking, Biome on changed TypeScript and production build passed. Real Redis-client outage regressions cover delayed startup and loss of an established connection. PostgreSQL 17.4 migration checks preserved existing rows, accepted 442-character values and passed when reapplied.

Production already received the bounded two-column repair after a verified backup and row-content/heap comparison of all 11,341 existing rows. A same-image backend rollout refreshed prepared statements; the bot then saved 1,141 previously blocked users. The migration records that repair for normal deployments. Source changes are not yet deployed. Do not narrow these columns during rollback or run unrelated pending migrations as an incident repair.

The PostgreSQL memory-retention cause remains unresolved. A proposed Bun idle timeout killed an active query in a local test and was excluded from this change.

Related incident PRs: telegram, polinetwork-cd, terraform.

@coderabbitai

coderabbitai Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: f6859e0a-17bf-4b3f-9ba2-c312225a9a56

📥 Commits

Reviewing files that changed from the base of the PR and between c78ec65 and af87b2d.

📒 Files selected for processing (9)
  • drizzle/0019_encrypted_unicode_names.sql
  • drizzle/meta/0019_snapshot.json
  • drizzle/meta/_journal.json
  • scripts/sql/20260917-widen-encrypted-user-names.sql
  • scripts/sql/README.md
  • src/db/schema/tg/users.ts
  • src/redis.ts
  • tests/cipher.test.ts
  • tests/redis-recovery.test.ts

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


Walkthrough

The pull request widens encrypted Telegram name columns to 512 characters and adds migration and repair support. It also changes Redis reconnect handling to continue retrying after exhausted attempts and adds recovery tests.

Changes

Encrypted Unicode name capacity

Layer / File(s) Summary
Schema and migration update
src/db/schema/tg/users.ts, drizzle/0019_encrypted_unicode_names.sql, drizzle/meta/0019_snapshot.json, drizzle/meta/_journal.json
Telegram first_name and last_name use varchar(512). Migration metadata records the updated schema.
Repair SQL and deployment documentation
scripts/sql/20260917-widen-encrypted-user-names.sql, scripts/sql/README.md
The repair changes only the two name columns with bounded lock and statement timeouts. Documentation records deployment and restart requirements.
Encrypted name capacity test
tests/cipher.test.ts
The test verifies Unicode encryption round trips and checks the column capacity against the encrypted payload length.

Redis reconnect recovery

Layer / File(s) Summary
Persistent reconnect strategy
src/redis.ts
Redis reconnects with capped exponential backoff and jitter. The previous retry limit and openSuccess state were removed.
Recovery validation
tests/redis-recovery.test.ts
Tests cover recovery when Redis becomes available after initial retries and when an established connection returns after an outage.

Sequence Diagram(s)

sequenceDiagram
  participant RedisClient
  participant reconnectStrategy
  participant RedisServer
  RedisClient->>RedisServer: connect
  RedisServer-->>RedisClient: connection unavailable
  RedisClient->>reconnectStrategy: request retry delay
  reconnectStrategy-->>RedisClient: exponential delay with jitter
  RedisClient->>RedisServer: reconnect
  RedisServer-->>RedisClient: HELLO and RPUSH responses
Loading

Priority: ➖ Normal

Change: Bug fix

Merge Risk: ⚪ Minimal · up to af87b

The widened name columns safely accommodate validated encrypted names, and no actionable current-head risk remains.

🚥 Pre-merge checks | ✅ 3 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 4 files. (5 skipped: 5 … Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly and concisely summarizes both primary changes: Redis connection recovery and preservation of encrypted names.
Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 4 files. (5 skipped: 5 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant