A production-style Retrieval-Augmented Generation (RAG) system for querying trusted AI security, red-teaming, and AI governance documentation.
SentinelRag combines dense vector retrieval, BM25 lexical retrieval, and Reciprocal Rank Fusion (RRF) to improve context retrieval before passing the selected documents to an LLM for grounded response generation.
The system uses FastAPI for the API layer, Redis Queue (RQ) for asynchronous processing, Qdrant Cloud for vector search, and LangSmith for tracing, evaluation, and observability.
- Implemented hybrid retrieval combining dense vector search and BM25 lexical retrieval.
- Implemented Reciprocal Rank Fusion (RRF) to merge independent retrieval rankings.
- Migrated vector infrastructure from local Qdrant to Qdrant Cloud, improving retrieval performance.
- Built an asynchronous RAG execution pipeline using FastAPI, Redis, and RQ workers.
- Integrated LangSmith tracing and LLM-as-a-Judge evaluation across correctness, relevance, retrieval relevance, and groundedness.
- Benchmarked the system at 11.88s P50 and 16.36s P99 latency across 20 evaluation runs.
┌──────────────────┐
│ User Query │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ FastAPI │
│ REST API │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ Redis Queue │
│ (RQ) │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ RQ Worker │
└────────┬─────────┘
│
┌─────────────┴─────────────┐
│ │
▼ ▼
┌───────────────┐ ┌───────────────┐
│ Dense │ │ BM25 │
│ Retrieval │ │ Retrieval │
│ Qdrant Cloud │ │ Lexical │
└───────┬───────┘ └───────┬───────┘
│ │
└─────────────┬─────────────┘
│
▼
┌──────────────────┐
│ RRF Reranking │
│ Reciprocal Rank │
│ Fusion │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ Retrieved Context│
└────────┬─────────┘
│
▼
┌──────────────────┐
│ LLM │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ Grounded Answer │
│ + Page Citations │
└──────────────────┘
- Hybrid RAG architecture
- Dense semantic retrieval with Qdrant Cloud
- Lexical retrieval using BM25
- Reciprocal Rank Fusion (RRF) for combining retrieval rankings
- Multi-document AI security knowledge base
- PDF ingestion and chunking pipeline
- Embedding-based document indexing
- Grounded responses with source/page citations
- Asynchronous processing using Redis Queue
- FastAPI REST API
- LangSmith tracing and observability
- Automated RAG evaluation pipeline
- LLM-as-a-Judge evaluation
- Retrieval and generation latency benchmarking
- Modular application architecture
SentinelRag uses two complementary retrieval strategies.
Queries are embedded and searched against document vectors stored in Qdrant Cloud.
Dense retrieval is useful for identifying passages that are semantically related to the query even when the exact terminology differs.
Query
│
▼
Embedding Model
│
▼
Qdrant Cloud
│
▼
Top-K Semantic Results
The query is also processed using BM25 lexical retrieval.
BM25 provides strong lexical matching for:
- Security terminology
- Acronyms
- Technical keywords
- Named frameworks
- Exact phrases
- Domain-specific terminology
Query
│
▼
BM25
│
▼
Top-K Lexical Results
The dense and BM25 result sets are merged using Reciprocal Rank Fusion (RRF).
Dense Results ─────┐
│
├──► RRF ──► Unified Ranking
│
BM25 Results ──────┘
This allows the retrieval layer to combine semantic similarity with lexical relevance before constructing the final context.
The initial implementation used local Qdrant storage.
The vector store was subsequently migrated to Qdrant Cloud, replacing the local deployment and improving retrieval performance while providing managed vector infrastructure.
Current retrieval architecture:
Query
│
┌────────┴────────┐
│ │
▼ ▼
Qdrant Cloud BM25
Dense Search Lexical Search
│ │
└────────┬────────┘
│
▼
RRF
│
▼
Retrieved Context
SentinelRag indexes a collection of AI security, red-teaming, and governance documentation stored in:
app/data/
| Document | Focus |
|---|---|
| SAFE-AI Full Report | AI safety and security |
| Solutions Landscape: Red-Teaming Taxonomy v1.0 | AI red-teaming methodologies and taxonomy |
| State of Agentic AI Security and Governance v2.01 | Security and governance of agentic AI systems |
| NIST AI 100-1 | AI-related standards and guidance |
Documents are processed through the following pipeline:
PDF Documents
│
▼
Document Loading
│
▼
Chunking
│
▼
Embedding Generation
│
▼
Qdrant Cloud Index
The documents are also made available to the lexical retrieval layer for BM25 search.
Run the ingestion pipeline whenever the document collection changes.
After retrieval and RRF ranking, the highest-ranked document chunks are passed to the generation stage.
User Query
│
▼
Hybrid Retrieval
│
▼
RRF Ranking
│
▼
Top Context
│
▼
Prompt Construction
│
▼
LLM
│
▼
Grounded Response
│
▼
Page Citations
The generation pipeline is designed to answer questions using the retrieved documentation rather than relying solely on the model's parametric knowledge.
SentinelRag uses Redis Queue (RQ) to decouple the API layer from the RAG execution pipeline.
Client
│
▼
FastAPI
│
▼
Redis Queue
│
▼
RQ Worker
│
├── Retrieval
├── RRF
├── Context Construction
├── LLM Invocation
└── Response Generation
This allows the API to return a Job ID while the RAG pipeline executes asynchronously.
Multiple workers can be started to support increased workload.
LangSmith is integrated into the application for tracing and evaluation.
It provides visibility into:
- RAG execution traces
- Retrieval steps
- LLM calls
- Evaluation results
- Latency
- Experiment runs
- RAG quality metrics
SentinelRag includes an automated evaluation pipeline using a custom benchmark dataset and LLM-based evaluators.
| Metric | Purpose |
|---|---|
| Correctness | Evaluates whether the generated answer correctly answers the question |
| Answer Relevance | Measures relevance of the generated answer to the query |
| Retrieval Relevance | Evaluates whether retrieved context is relevant to the query |
| Groundedness | Determines whether the answer is supported by the retrieved documentation |
| Latency | Measures end-to-end pipeline latency |
Across 20 evaluation runs:
Correctness 1.00 AVG
Answer Relevance 1.00 AVG
Retrieval Relevance 1.00 AVG
Faithfulness 31% Grounded
Latency
P50 11.88s
P99 16.36s
The evaluation pipeline is integrated with LangSmith for experiment tracking and trace-level analysis.
├── README.md
├── app
│ ├── data
│ │ ├── SAFEAI_Full_Report.pdf
│ │ ├── Solutions-Landscape-Red-Teaming-Taxonomy-v.1.0.pdf
│ │ ├── State-of-Agentic-AI-Security-and-Governance-v2.01.pdf
│ │ └── nist.ai.100-1.pdf
│ ├── evaluation
│ │ ├── dataset.csv
│ │ ├── evaluate.py
│ │ └── metrics
│ │ ├── answer_relevance.py
│ │ ├── correctness.py
│ │ ├── faithfulness.py
│ │ ├── llm_client.py
│ │ └── retrieval_relevance.py
│ ├── rag
│ │ ├── clients
│ │ │ ├── qdrant_client.py
│ │ │ └── ragqueue_client.py
│ │ ├── config
│ │ │ └── config.py
│ │ ├── ingestion
│ │ │ └── index.py
│ │ ├── main.py
│ │ ├── queues
│ │ │ └── worker.py
│ │ └── server.py
│ ├── reports
│ │ ├── dataset_92849185-c26e-4cb7-9975-1d207824945f.csv
│ │ └── rag-doc-relevance-87cf829a.csv
│ └── requirements.txt
├── docker-compose.yml
- Python
- FastAPI
- LangChain
- Qdrant Cloud
- BM25
- Reciprocal Rank Fusion (RRF)
- Embedding Models
- LLM APIs
- Groq
- NVIDIA API
- Redis
- Redis Queue (RQ)
- Docker
- LangSmith
- Pandas
Make sure you have:
- Python 3.x
- Docker
- Redis
- Qdrant Cloud account
- Groq API key
- NVIDIA API key
- LangSmith API key
git clone https://github.com/AllenGeorge08/SentinelRag.git
cd SentinelRagpython3 -m venv venvsource venv/bin/activatevenv\Scripts\activatepip install -r requirements.txtIf your dependencies are maintained inside app/requirements.txt, use:
cd app
pip install -r requirements.txtCreate a .env file based on .env.example.
cp .env.example .envThe current environment configuration is:
GROQ_API_KEY=
NVIDIA_API_KEY=
LANGSMITH_API_KEY=
QDRANT_API_KEY=
QDRANT_CLUSTER_ENDPOINT=Fill in each value with the corresponding API credential/configuration.
| Variable | Purpose |
|---|---|
GROQ_API_KEY |
Authentication for Groq-hosted models |
NVIDIA_API_KEY |
Authentication for NVIDIA-hosted models |
LANGSMITH_API_KEY |
LangSmith tracing and evaluation |
QDRANT_API_KEY |
Authentication for Qdrant Cloud |
QDRANT_CLUSTER_ENDPOINT |
Qdrant Cloud cluster endpoint |
Never commit your .env file or API keys to version control.
Start the required local infrastructure:
docker compose up -dThe application uses Redis as the backing queue for RQ workers.
Qdrant is hosted on Qdrant Cloud and therefore does not need to be started locally.
Navigate to the application directory:
cd appRun:
python -m rag.ingestion.indexThe ingestion process:
- Loads PDFs from
app/data - Splits documents into chunks
- Generates embeddings
- Uploads vectors to Qdrant Cloud
- Prepares documents for lexical retrieval
Run this whenever the knowledge base is modified.
Open a new terminal and activate the virtual environment.
From the application directory:
cd app
rq workerThe worker handles the complete RAG execution pipeline:
Job
│
▼
Hybrid Retrieval
│
▼
RRF
│
▼
Context Construction
│
▼
LLM
│
▼
Response
You can run multiple workers for concurrent job processing.
Open another terminal:
cd app
python -m rag.mainThe API will be available at:
http://localhost:8000
Interactive Swagger documentation:
http://localhost:8000/docs
Open the Swagger interface:
http://localhost:8000/docs
Send a request to:
POST /chat
The API returns a Job ID.
Use:
GET /job-status
with the Job ID to retrieve the result once processing is complete.
The generated response includes the answer along with document/page citations where applicable.
The evaluation pipeline can be executed independently from the application.
From the repository root:
python -m app.evaluation.evaluateThe evaluation process:
Benchmark Dataset
│
▼
RAG Pipeline
│
▼
Generated Answers
│
├────► Correctness
├────► Answer Relevance
├────► Retrieval Relevance
├────► Groundedness
└────► Latency
│
▼
LangSmith
The experiment and associated traces are uploaded to LangSmith.
Latest evaluation results:
Current benchmark results across 20 evaluation runs:
| Metric | Result |
|---|---|
| Correctness | 1.00 AVG |
| Answer Relevance | 1.00 AVG |
| Retrieval Relevance | 1.00 AVG |
| P50 Latency | 11.88s |
| P99 Latency | 16.36s |
These measurements represent the current implementation and benchmark configuration.
Dense retrieval is effective at semantic matching but can underperform on exact technical terminology.
BM25 complements this by providing lexical matching.
Combining both through RRF provides a retrieval strategy that considers both:
Semantic Similarity
+
Lexical Similarity
↓
RRF Ranking
↓
Better Context
The initial implementation used local Qdrant storage. Migrating to Qdrant Cloud provided managed vector infrastructure and improved retrieval performance compared with the local setup.
RAG execution involves multiple potentially expensive operations, including retrieval, prompt construction, and LLM inference.
RQ separates these operations from the HTTP request lifecycle, allowing FastAPI to operate as the request-facing layer while workers execute background jobs.
RAG quality cannot be evaluated reliably through generation quality alone.
LangSmith provides visibility into:
Retrieval
+
Generation
+
Evaluation
+
Latency
+
Traces
This makes it possible to inspect both what the system retrieved and what the model generated from that context.
Potential next steps:
- Cross-encoder reranking
- Query rewriting
- Query decomposition
- Metadata-aware filtering
- Streaming responses
- Multi-turn conversation support
- Conversation memory
- Larger evaluation benchmark
- Automated evaluation regression tests
- CI/CD integration
- Containerized production deployment
- Retrieval and generation latency optimization
This project is intended for educational, research, and demonstration purposes.