feat: add liquidity reservation layer for concurrent loan creation (#81)

[LaPoshBaby]

feat: add liquidity reservation layer for concurrent loan creation (#81)

Summary

Closes #81. Introduces a Redis-backed reservation ledger that prevents concurrent loan creations from over-committing shared pool funds between credit assessment and on-chain settlement.

Problem Statement

The current implementation of loans.service.createLoan() has a critical race condition in its loan creation flow:

1. Availability Check: The service reads pool availability from the database
2. Transaction Building: It builds the funding XDR (Stellar transaction)
3. User Signing: Returns the XDR for the user to sign
4. On-chain Settlement: User submits the signed transaction to the blockchain

The Race Condition: Between steps 1 and 4, there is no commitment of funds. Two concurrent callers can:

• Both pass the availability check against the same pool funds
• Both receive valid funding XDRs to sign
• The first transaction succeeds on-chain
• The second transaction reverts with InsufficientLiquidity error

Impact:

• Users waste transaction fees signing transactions that will fail
• Dangling provisional loans remain in the database
• Poor user experience and potential support overhead
• No guarantee of liquidity availability during high-concurrency periods

Solution Overview

Implement a per-pool reservation ledger that atomically reserves liquidity during the loan creation process, ensuring that concurrent requests cannot over-commit available funds.

Architecture

The solution introduces a new ReservationsModule with the following components:

1. Reservation Store Interface

• Abstract interface (ReservationStore) defining reservation operations
• Two implementations:
  • RedisReservationStore: Production implementation using Redis with Lua scripts for atomicity
  • InMemoryReservationStore: Development/testing fallback using in-memory maps

2. Liquidity Reservation Service

• Core business logic for acquiring, releasing, and managing reservations
• Integrates with the loan creation flow
• Handles TTL expiration and cleanup

3. Transaction Status Checker Integration

• Hooks into the existing transaction monitoring job
• Automatically releases reservations on settlement events:
  • FUNDED: Transaction succeeded - release reservation
  • FAILED: Transaction reverted - release reservation
  • LOAN_REPAY: Loan repaid - release reservation
  • TTL: Reservation expired - release automatically

4. Drift Reconciliation

• Cron job running every 5 minutes (@Cron('0 */5 * * * *'))
• Compares total reserved amounts against on-chain locked_liquidity
• Flags positive drift (reservations > locked liquidity) via Prometheus gauge
• Provides operational visibility into reservation accuracy

5. Metrics and Observability

Comprehensive Prometheus metrics:

• stepfi_liquidity_reservations_inflight: Current number of active reservations (gauge)
• stepfi_liquidity_reservations_acquired_total: Total reservations acquired (counter)
• stepfi_liquidity_reservations_released_total: Total reservations released by reason (counter with labels)
• stepfi_liquidity_reservations_rejected_total: Total reservations rejected by reason (counter with labels)
• stepfi_liquidity_reservations_drift_stroops: Drift amount in stroops (gauge)
• stepfi_liquidity_reservations_reconciliations_total: Total reconciliation runs (counter)
• stepfi_liquidity_reservations_ttl_expirations_total: Total TTL expirations (counter)

Technical Implementation Details

Reservation Acquisition Flow

1. Mutex Acquisition: Per-pool mutex ensures serialized access to reservation logic
2. Capacity Check: Verifies pool has sufficient available liquidity
3. Atomic Reservation: Uses Redis Lua scripts or in-memory atomic operations
4. TTL Assignment: Default 15-minute expiration (configurable via env var)
5. Reservation ID Generation: Unique identifier for tracking
6. Return to Caller: Reservation metadata included in loan creation response

Reservation Release Flow

Reservations are released in the following scenarios:

• Successful Settlement: Transaction confirmed as FUNDED
• Failed Transaction: Transaction reverted with FAILED status
• Loan Repayment: Loan fully repaid (LOAN_REPAY)
• TTL Expiration: Automatic cleanup after 15 minutes
• Manual Release: Administrative operations (future enhancement)

Redis Implementation Details

Lua Scripts for Atomicity:

• acquire.lua: Atomically checks capacity and creates reservation
• release.lua: Atomically removes reservation and updates metrics
• Scripts ensure consistency across Redis cluster deployments

Data Structure:

• Key pattern: reservation:{poolId}:{reservationId}
• Fields: amount, userId, loanId, createdAt, expiresAt, status
• TTL: Set automatically on key creation

In-Memory Fallback

• Activated when LIQUIDITY_RESERVATION_REDIS_URL is unset
• Uses JavaScript Map with mutex locks
• Suitable for development and testing environments
• Not recommended for production (no persistence across restarts)

Database Schema Changes

New columns added to loans table:

• reservation_id: UUID referencing the reservation
• reservation_expires_at: Timestamp for TTL tracking
• Indexes added for efficient querying

API Changes

CreateLoanResponseDto Enhancement

Three new optional fields added to the response:

• reservationId: Unique identifier for the reservation
• expiresAt: ISO 8601 timestamp when reservation expires
• poolCapacityUsd: Current available capacity in USD

Backwards Compatibility: Existing clients continue to work as these fields are optional. The core contract (returning funding XDR) remains unchanged.

Configuration

Environment Variables

New environment variables added to docs/setup/environment-variables.md:

• LIQUIDITY_RESERVATION_REDIS_URL: Redis connection string (optional, defaults to in-memory)
• LIQUIDITY_RESERVATION_TTL_SECONDS: Reservation TTL in seconds (default: 900)
• LIQUIDITY_RESERVATION_ENABLED: Feature flag (default: true)

Testing

Unit Tests

New test suite: test/unit/modules/liquidity/reservations/liquidity-reservation.service.spec.ts

• Tests reservation acquisition and release
• Tests TTL expiration logic
• Tests concurrent access scenarios
• Tests capacity enforcement
• Tests metrics emission

Integration Tests

Updated existing loan service tests:

• test/unit/modules/loans/loans.service.spec.ts
• test/unit/modules/loans/loans.controller.spec.ts

Concurrency Test

New test specifically validates the race condition fix:

• Simulates 10 concurrent loan creation requests
• Pool capacity exactly fits 5 loans
• Asserts exactly 5 succeed, 5 are rejected
• Proves no double-spend under parallel load

Transaction Status Checker Tests

Rewritten test suite: test/unit/jobs/transaction-status-checker/transaction-status-checker.service.spec.ts

• Tests reservation release on various transaction statuses
• Tests integration with reservation service
• Tests error handling scenarios

Verification Results

• Build: npm run build - Clean, zero TypeScript errors
• Tests: npm test - 303/303 tests pass across 26 suites (no regression)
• Linting: npx eslint scoped to changed files - Clean, no violations

Acceptance Criteria Validation

✅ Concurrent loan creations cannot over-commit available liquidity

• Per-pool mutex ensures serialized access
• Atomic reservation operations prevent race conditions

✅ Reservation acquired before funding XDR is returned

• reserveForLoan() called before buildCreateLoanTransaction() in loans.service.ts

✅ Reservations expire (TTL) and release on failure/settlement

• 15-minute default TTL with automatic cleanup
• Transaction status checker releases on settlement events

✅ Reservation totals reconciled against on-chain locked liquidity

• Cron job runs every 5 minutes
• Drift metrics emitted for monitoring

✅ Concurrency test proves no double-spend under parallel requests

• New test asserts 5/10 split when capacity fits exactly 5

Risk Assessment and Rollback Plan

Risks

1. Redis Unavailability

   • Mitigation: Fails closed with 503 error rather than overcommitting
   • Fallback: In-memory store for dev/test environments

2. Reservation Drift

   • Mitigation: Reconciliation cron emits metrics before user-visible impact
   • Monitoring: Prometheus alerts on positive drift

3. TTL Expiration Edge Cases

   • Mitigation: Transaction status checker releases on settlement
   • Monitoring: TTL expiration metrics track cleanup rate

Rollback Procedure

If issues arise, rollback involves:

1. Remove ReservationsModule from src/app.module.ts
2. Revert changes to src/modules/loans/loans.service.ts
3. Remove reservation-related columns from database (or leave unused)
4. This returns the system to pre-core: add liquidity reservation and locking for concurrent loan creation #81 behavior

Backwards Compatibility

• Controller Contract: Unchanged - createLoan still returns funding XDR
• Response DTO: Three new optional fields added, existing fields unchanged
• Database: New columns added with null defaults, no breaking changes
• Configuration: New environment variables are optional with sensible defaults

Performance Considerations

• Redis Latency: ~1-2ms per reservation operation
• Mutex Contention: Per-pool mutex minimizes contention across pools
• Memory Usage: In-memory store uses ~100 bytes per reservation
• Cron Overhead: Reconciliation runs every 5 minutes, minimal impact

Future Enhancements

Potential improvements for future iterations:

• Administrative API for manual reservation release
• Reservation extension mechanism for long-running transactions
• Historical reservation analytics and reporting
• Multi-pool reservation support for complex loan structures
• Circuit breaker for Redis unavailability scenarios

[EmeditWeb]

Review verdict: ✅ APPROVE (strong)

Well-architected concurrency control for the reserve-then-settle race. Highlights:

• Lua-atomic Redis operations (ACQUIRE_RESERVATION_LUA / RELEASE_RESERVATION_LUA) — the correct way to make the ZSET + metadata moves atomic under concurrency.
• Reserve-before-settle with rollback in loans.service — reserves liquidity before building the funding XDR and releases on failure (releaseReservationQuietly).
• @Cron('0 */5 * * * *') cleanup (not BullMQ — matches the repo rule), no any, dual in-memory/Redis store behind an interface, evalsha→eval NOSCRIPT fallback. CI green.

Notes before merge:

1. Confirm the added Redis ZSET + Lua-per-reservation load fits the Upstash free-tier budget (the reason BullMQ was removed). The in-memory fallback suggests Redis is optional/configurable — please document the intended production mode.
2. Ensure the new reservation_id field on the create-loan response DTO has an @ApiProperty decorator.

Review-depth note: I verified the atomicity design, standards, and reserve/rollback integration path — not every branch of the Lua scripts. Given this guards pooled funds, one focused pass over the two Lua scripts before merge would be worthwhile. Otherwise this is approve-quality.