VectorLens: High-Performance Observable RAG Engine

A production-grade, highly observable Retrieval-Augmented Generation (RAG) platform and vector database exploration system. VectorLens pairs an asynchronous Python/FastAPI backend with a modern Next.js dashboard to deliver sub-15ms semantic caching, custom approximate nearest neighbor (ANN) routing, cross-encoder reranking, and decoupled system evaluation.

---

📌 Project Overview

Most basic RAG prototypes struggle with two primary enterprise hurdles: unpredictable generation latency and unobservable retrieval pipelines.

VectorLens solves these bottlenecks by implementing:

• Multi-Engine Vector Retrieval — Ingests document corpora into custom ANN graph architectures (HNSW) alongside baseline search methods.
• Hashed Semantic Cache Layer — Uses cloud-hosted Upstash Redis to intercept repeated and structurally similar queries, reducing retrieval latency from ~1.2s–2.4s down to ~14ms–115ms (~95% latency reduction).
• Dynamic Dataset Versioning — Auto-invalidates stale cache keys whenever new document chunks are indexed into the engine, preventing stale data generation.
• Decoupled Offline Evaluator — An isolated background benchmarking worker that computes Recall@K and latency over MS-MARCO validation slices and persists report cards to MongoDB without affecting live API throughput.
• Engine Inspector (Full-Trace Telemetry) — A real-time observability panel providing granular trace execution, routing sources, token metrics, and cache telemetry directly in the UI.

---

🛠️ Architecture & Tech Stack

[ Client: Next.js App Router (Tailwind + Recharts) ]
                    │
                    ▼  HTTP / REST
     [ FastAPI Async Gateway (Python 3.12) ]
        │                        │
        ├── Ingestion Pipeline   └── Query Pipeline
        │     │                        │
        │     ▼                        ▼
        │   PyPDF Extraction         Query Rewrite Layer
        │     │                        │
        │     ▼                        ▼
        │   Recursive Chunking       Upstash Redis Cache Check (v_current)
        │     │                        │
        │     ▼                  [Cache Hit] ──► Instant Return (~14ms)
        │   768D Embeddings            │
        │     │                  [Cache Miss]
        │     ▼                        │
        │   HNSW Index Persistence     ├─► HNSW ANN Search (Top-15)
        │                              ├─► Cross-Encoder Rerank (Top-5)
        │                              ├─► Groq Llama-3 Generation
        │                              └─► Cache Save & Telemetry Record
        ▼
[ Data Layers: MongoDB Atlas (Metadata & Benchmarks) + Upstash Redis (Cache & Versions) ]

Layer	Technologies
Frontend	Next.js (App Router), React, Tailwind CSS, Lucide Icons, Recharts
Backend API	FastAPI (Async ASGI), Uvicorn, Pydantic
Embeddings & Inference	all-mpnet-base-v2 (768-dimensional vector space), Groq Llama 3.1 (llama-3.1-8b-instant)
Reranker Engine	HuggingFace ms-marco-MiniLM-L-6-v2 Cross-Encoder
Caching & Version State	Upstash Redis (Hashed query caches with TTL + incremental version control)
Persistence & Telemetry	MongoDB Atlas (Document chunk metadata, engine configurations, evaluation benchmark history)

---

🚀 Key Features Implemented

1. Ingestion Pipeline & Document Management

• PDF Processing — Drag-and-drop file ingestion via Next.js Knowledge Base interface.
• Recursive Chunking — Automated document segmenting (500-character windows with 50-character overlaps).
• State Versioning — Increments the global system:dataset_version in Redis upon file upload to segregate cached responses across data updates.
• Metadata Persistence — Stores processed document size, indexed chunk counts, filenames, and upload timestamps in MongoDB Atlas.

2. Observable Query Playground & RAG Routing

• Configurable Vector Routing — Supports algorithmic routing switches (HNSW, KD-Tree, Brute Force).
• Engine Inspector — Live breakdown of query execution stages:
  • Query Rewriter trace
  • 768D embedding generation
  • HNSW ANN search phase
  • Cross-Encoder scoring pass
  • LLM token generation
• Dual-State Latency Tracking — Visual tagging showing whether data came from Upstash Redis (cache hit) or HNSW Index → Llama-3 (cold generation).

3. Asynchronous Benchmarking Framework (evaluator.py)

• Non-Blocking Execution — Runs as an isolated worker to evaluate system performance on MS-MARCO sample validation slices.
• Comparative Metrics — Evaluates Baseline (HNSW only) against Advanced (HNSW + Cross-Encoder Rerank) for:
  • Recall@5
  • Average Query Latency (ms)
  • Relative Accuracy Improvement
• Decoupled Database Sync — Writes evaluation runs directly into MongoDB Atlas, allowing the /evaluation UI to refresh instantly without triggering expensive model runs on demand.

4. Telemetry Dashboard

• Displays live semantic cache hit rate, total vector corpus count, and cumulative queries processed.
• Visualizes global latency trends using Recharts and provides an active simulated server log stream.

---

📂 Repository Structure

vectorlens/
├── backend/
│   ├── app/
│   │   ├── api/
│   │   │   └── routes.py         # REST endpoints (/upload, /ask, /metrics, /evaluation)
│   │   ├── core/
│   │   │   └── database.py       # Motor async MongoDB Atlas connection
│   │   ├── engine/
│   │   │   ├── hnsw.py           # Custom HNSW Graph implementation
│   │   │   ├── kdtree.py         # KD-Tree indexing baseline
│   │   │   └── brute_force.py    # Brute-force linear scan baseline
│   │   └── services/
│   │       ├── ingestion.py      # PDF parsing, chunking, and embedding logic
│   │       └── rag.py            # Retrieval pipeline, reranker, and Groq LLM integration
│   ├── evaluator.py              # Standalone background benchmarking worker
│   ├── main.py                   # FastAPI server entry point
│   └── requirements.txt          # Backend dependencies
│
├── frontend/
│   ├── app/
│   │   ├── dashboard/page.tsx      # Live telemetry and KPI monitoring
│   │   ├── knowledge-base/page.tsx # PDF upload and document metadata grid
│   │   ├── playground/page.tsx     # Interactive chat, RAG inspector, and trace drawer
│   │   └── evaluation/page.tsx     # MS-MARCO benchmark comparison reports
│   ├── components/                 # Shared UI elements (Sidebar, Navbar, Metrics cards)
│   └── package.json
└── README.md

---

⚙️ Local Development Setup

1. Prerequisites

• Python 3.11+
• Node.js 18+ and npm/pnpm
• MongoDB Atlas cluster URL
• Upstash Redis database URL
• Groq Cloud API key

2. Backend Setup

cd backend
python -m venv venv

# Windows
venv\Scripts\activate
# Linux/macOS
source venv/bin/activate

pip install -r requirements.txt

Create a .env file in backend/:

MONGODB_URI=your_mongodb_connection_string
UPSTASH_REDIS_URL=your_upstash_redis_url
GROQ_API_KEY=your_groq_api_key

Start the FastAPI application:

uvicorn main:app --reload --port 8000

3. Frontend Setup

cd ../frontend
npm install

Create a .env.local file in frontend/:

NEXT_PUBLIC_API_URL=http://127.0.0.1:8000

Start the Next.js development server:

npm run dev

Visit http://localhost:3000 to interact with the engine.

---

🚧 Current Status & Roadmap

• Monorepo restructuring and git commit workflow
• Custom vector search engines (Brute Force, KD-Tree, HNSW Graph)
• Ingestion pipeline with persistent versioning in Redis
• Next.js frontend with dynamic Knowledge Base, Dashboard, and Evaluation tabs
• Decoupled evaluation engine writing real analytics reports to MongoDB
• Active Focus: Synchronizing live HNSW index graph persistence across server restarts to eliminate cold retrieval empty context responses.
• Next Up: Upgrading query-hash caching to vector similarity-based embedding threshold caching in Redis.