Enterprise AI Knowledge Platform (RAG)
Production RAG platform with hybrid search & granular access control
Problem & Architectural Solution
Core business challenges, domain requirements, and technical strategy
The Problem & Bottlenecks
Internal technical teams spent excessive time manually cross-referencing dispersed architectural specs, incident postmortems, and compliance protocols. Standard vector-only RAG pipelines frequently produced false positives on exact technical terms (error codes, internal acronyms) and lacked granular data permission boundaries.
The Implemented Solution
Architected an asynchronous FastAPI backend integrating a dual-stage retrieval engine: semantic dense embeddings (OpenAI text-embedding-3-small) indexed via HNSW in pgvector, combined with PostgreSQL full-text tsvector search. Injected row-level security tokens into SQL execution contexts to guarantee that generated contexts only contain passages authorized for the querying user.
Detailed Scope & Objectives
The Enterprise AI Knowledge Platform was engineered to solve knowledge fragmentation across multi-department enterprise repositories. By pairing PostgreSQL 17's pgvector extension with BM25 lexical keyword matching through Reciprocal Rank Fusion (RRF), the system delivers precise, source-attributed responses while strictly enforcing document-level role-based access controls.
System Architecture & Data Flow
Component boundaries, async pipeline execution, and transaction lifecycle
Document Ingestion & Context-Aware Chunking
Hierarchical markdown parser segments documents along semantic heading boundaries with dynamic 15% overlap, preventing fragment truncation and generating 1536-dimensional embeddings.
Hybrid Vector + Lexical Search
Executes parallel queries: pgvector cosine distance over HNSW indexes and PostgreSQL tsvector/tsquery lexical search, combining candidate scores using Reciprocal Rank Fusion (RRF).
Security Scope & RBAC Filtering
Enforces tenant and department security scopes directly in the SQL WHERE clause, mathematically guaranteeing unauthorized records never reach the context window.
Grounded LLM Generation & Citation Streaming
Constructs a grounded system prompt with strict citation constraints and streams synthesized tokens to the client via FastAPI Server-Sent Events (SSE).
flowchart TD
subgraph Ingestion [Ingestion Pipeline]
Doc[Raw Docs / Markdown] --> Chunk[Semantic Markdown Splitter]
Chunk --> Embed[Embedding Generator: text-embedding-3-small]
Embed --> PG[(PostgreSQL 17 + pgvector)]
end
subgraph Query [Real-Time Retrieval & Generation]
User((User Query)) --> API[FastAPI Async Endpoint]
API --> Auth{JWT & RBAC Check}
Auth --> Cache{Redis Query Cache}
Cache -- Hit --> StreamOut[SSE Token Stream]
Cache -- Miss --> Hybrid[Hybrid Search Engine]
Hybrid -->|HNSW Vector Sim| PG
Hybrid -->|BM25 Lexical tsvector| PG
PG --> RRF[Reciprocal Rank Fusion]
RRF --> LLM[LLM Synthesis with Citation Prompt]
LLM --> StreamOut
endTechnology Stack Breakdown
Explicit technical responsibilities and tooling justification per layer
Backend & Systems
High-concurrency async REST API and Server-Sent Events streaming
Async runtime powering document ingestion and retrieval logic
Document parsing, node management, and prompt assembly framework
Database & Caching
Primary ACID transactional database and document metadata store
In-database vector embeddings with HNSW indexing for sub-50ms search
Embedding cache and frequent query synthesis response caching
AI & Model Pipelines
Dense semantic vector representations (1536 dims)
Context-grounded synthesis with structured source citation
Infrastructure & Security
Reproducible multi-container local and staging environments
Stateless claims-based authentication and department role mapping
Frontend & Client
Interactive query explorer with streaming markdown renderer
High-contrast technical dark mode UI with citation badges
Deployment & Runtime
ASGI process manager with asynchronous worker loops
Optimized containerized production workload execution
Key Technical Capabilities
Production-grade features, system subsystems, and upcoming roadmap items
Hybrid BM25 + Vector Retrieval
CompletedCombines dense semantic understanding with exact lexical matching to eliminate technical acronym blind spots.
PostgreSQL pgvector HNSW Indexing
CompletedLeverages Hierarchical Navigable Small World (HNSW) graphs inside PostgreSQL, avoiding standalone vector cluster overhead.
Document-Level RBAC Filtering
CompletedEnforces authorization at the database query level, preventing data leakage across department access tiers.
Redis Response & Embedding Caching
CompletedCaches identical semantic query vectors and answer tokens to achieve sub-50ms cache hits on common inquiries.
Automated Confidence & Citation Scoring
CompletedComputes similarity distance thresholds and generates verifiable inline document citation footnotes.
Multi-Format Ingestion Connectors (PDF, Confluence)
PlannedScheduled background workers to ingest binary PDFs, Word docs, and API documentation continuously.
Architecture Decisions & Trade-Offs
Technical context, decision rationales, and verified system outcomes
1PostgreSQL pgvector over Standalone Vector DB
Needed vector search capabilities without introducing cluster synchronization latency or duplicate authorization logic.
Selected pgvector within PostgreSQL 17 to keep relational SQL data, access roles, and embeddings in one ACID store.
Simplified backup and restore procedures, guaranteed transaction consistency, and cut cloud hosting overhead by 60%.
2Server-Sent Events (SSE) vs WebSockets for Streaming
Waiting for complete LLM responses caused high perceived latency (3-4 seconds before first text visible).
Implemented unidirectional SSE over HTTP/2 instead of bi-directional WebSocket connection management.
Reduced time-to-first-token to under 250ms with automatic HTTP reconnect handling and lower connection state overhead.
3Reciprocal Rank Fusion (RRF) for Search Merging
Raw vector cosine scores and BM25 relevance scores exist on different numerical scales, making naive addition ineffective.
Applied RRF algorithm with rank constant k=60 to merge ranking lists without score calibration dependencies.
Improved top-3 retrieval precision from 81% (vector-only) to 94.2% (hybrid RRF).
Engineering Challenges & Solutions
Real architectural bottlenecks encountered and the engineering rationale behind their resolution
#1Vector Search Hallucinations on Exact Technical Terms
Pure vector similarity frequently missed exact error codes (e.g. 'ERR_SOCKET_TIMEOUT_104') because embeddings map semantic concepts rather than literal characters.
Integrated PostgreSQL full-text search with customized tsvector dictionaries and combined rankings using Reciprocal Rank Fusion.
#2Multi-Tenant Authorization Leakage Risks
Filtering documents post-retrieval in Python application memory often resulted in empty context windows when top candidates were filtered out.
Pushed user role access arrays directly into the SQL WHERE clause before vector similarity ordering.
Security, Integrity & Reliability
Production safeguards, boundary enforcement, and fault-tolerance patterns
Granular Access Boundary Enforcement
Row-level security scopes ensure vector search queries only scan documents the requesting user's JWT grants access to.
Strict Prompt Sandboxing & Injection Mitigation
Retrieved context passages are sanitized and delimited using structured XML tags with explicit instruction-override guardrails.
Rate Limiting & Token Quotas
Redis token-bucket rate limiters prevent API abuse and control downstream LLM generation expenses per client tenant.
Verified Results & Status
Factual metrics, operational milestones, and current production state
- Delivered sub-400ms average retrieval latency across enterprise document corpuses exceeding 50,000 pages.
- Attained 94.2% top-3 retrieval precision score on domain-specific technical documentation benchmarks.
- Zero recorded cross-department authorization bypasses during automated security boundary penetration tests.
Lessons Learned & Retrospective
Key technical takeaways that inform future platform architecture decisions
Fixed-character chunking destroys tabular and structured code semantics; layout-aware section parsing is essential for high RAG accuracy.
Hybrid search is mandatory for technical domains where users query specific entity identifiers and error strings.
Streaming responses significantly improve user experience even when overall model synthesis takes 2-3 seconds.
RAG Complaint Chatbot
AI & RAG Engineering
Autonomous AI Agent Workflow Orchestrator
AI & Automation
Related Case Studies
Autonomous AI Agent Workflow Orchestrator
An autonomous agent execution platform that parses natural language operational goals into structured, deterministic multi-step API tool execution plans.
Christian Digital Content Platform - Backend API
A scalable RESTful backend API for a Christian digital content platform, providing authentication, role-based access control, Christian articles and devotionals, digital book catalog management, user libraries, and reading progress.