RAG Complaint Chatbot
Evidence-grounded AI analysis of 1.37 million consumer complaints
Problem & Architectural Solution
Core business challenges, domain requirements, and technical strategy
The Problem & Bottlenecks
The Consumer Financial Protection Bureau (CFPB) receives an immense volume of consumer complaints about financial products and services. Manually sifting through this dataset, which includes 1.37 million records, to find relevant information is extremely challenging due to its sheer scale, the unstructured nature of complaint narratives, and the diverse range of financial issues covered. Traditional keyword search often falls short, missing semantically related complaints and leading to slow, inefficient manual investigations.
The Implemented Solution
Developed a comprehensive RAG pipeline to automate and enhance the analysis of CFPB consumer complaints. The solution involves cleaning and chunking complaint narratives, converting them into 384-dimensional embeddings using Sentence Transformers, and storing these efficiently in a FAISS vector index. Analyst queries are semantically matched against this index to retrieve the most relevant complaint excerpts, which are then used to ground the answers generated by a FLAN-T5 language model. A Gradio interface provides an interactive way to query the system and view both the generated answers and their supporting evidence.
Detailed Scope & Objectives
The RAG Complaint Chatbot is an AI/ML engineering project focused on developing an evidence-grounded system for analyzing large volumes of consumer complaints from the CFPB database. This project demonstrates the full lifecycle of a Retrieval-Augmented Generation (RAG) pipeline, from data preparation and semantic indexing to efficient retrieval and grounded language generation. It aims to empower analysts to quickly extract insights and evidence from unstructured complaint narratives.
System Architecture & Data Flow
Component boundaries, async pipeline execution, and transaction lifecycle
Data Preparation
The CFPB consumer complaint dataset undergoes rigorous cleaning and filtering to extract useful complaint narratives, removing noise and standardizing formats for subsequent processing.
Semantic Representation
Each cleaned complaint text chunk is transformed into a 384-dimensional numerical vector (embedding) using the `sentence-transformers/all-MiniLM-L6-v2` model, capturing its semantic meaning.
Vector Indexing
The generated embeddings are efficiently stored and indexed using FAISS (Facebook AI Similarity Search), enabling rapid similarity queries across the large dataset.
Retrieval
When an analyst poses a question, it is first converted into an embedding. This query embedding is then matched against the FAISS index to semantically retrieve the most relevant complaint excerpts (evidence).
Grounded Generation
The retrieved complaint evidence is fed into a `google/flan-t5-small` language model. This model generates a concise answer that is 'grounded' or supported by the provided evidence, reducing hallucinations.
Evidence-Based Interface
A Gradio-based user interface presents the generated answer to the analyst along with the exact raw complaint excerpts that were used as evidence, ensuring transparency and trustworthiness.
flowchart TD
Data[CFPB Complaint Dataset]
Clean[Preprocessing & Filtering]
Chunk[Text Chunking]
Embed[Sentence Transformer Embeddings]
FAISS[(FAISS Vector Store)]
Query[Analyst Query]
Retrieve[Semantic Retrieval]
Context[Relevant Complaint Evidence]
LLM[FLAN-T5 Generation]
Answer[Grounded Answer]
UI[Gradio Interface]
Data --> Clean
Clean --> Chunk
Chunk --> Embed
Embed --> FAISS
UI --> Query
Query --> Retrieve
Retrieve --> FAISS
FAISS --> Context
Context --> LLM
Query --> LLM
LLM --> Answer
Answer --> UI
Technology Stack Breakdown
Explicit technical responsibilities and tooling justification per layer
Key Technical Capabilities
Production-grade features, system subsystems, and upcoming roadmap items
Large-Scale Complaint Indexing
CompletedProcessed and indexed 1.37 million CFPB consumer complaints, making them semantically searchable.
Semantic Retrieval
CompletedUses dense sentence embeddings and FAISS similarity search to retrieve semantically relevant complaint evidence, overcoming keyword limitations.
Grounded Answer Generation
CompletedGenerates concise answers from retrieved complaint context rather than relying solely on the language model's internal knowledge, improving factual accuracy.
Evidence Panel
CompletedThe user interface explicitly displays the complaint excerpts that support the generated response, enhancing transparency and analyst trust.
Retrieval Evaluation
CompletedAchieved a 100% retrieval rate on the documented benchmark, where the top-5 relevant chunks were found for all benchmark questions.
Architecture Decisions & Trade-Offs
Technical context, decision rationales, and verified system outcomes
1Sentence Transformers for Embeddings
Analyzing complaint narratives required semantic matching, as simple keyword search often missed relevant documents due to varied phrasing and informal language.
Employed `all-MiniLM-L6-v2` from Sentence Transformers to generate compact and semantically rich 384-dimensional vector representations of complaint texts.
Enabled highly effective semantic retrieval across a large corpus of diverse complaint narratives, improving the ability to find contextually relevant information.
2FAISS for Vector Retrieval
The project involved performing efficient similarity searches across a dataset of 1.37 million complaint records, requiring a fast and scalable vector indexing solution.
Utilized FAISS (Facebook AI Similarity Search) as the dense vector index for its optimized algorithms for high-dimensional vector search.
Achieved rapid local vector retrieval, enabling near real-time querying of the massive complaint dataset without the overhead of a separate, complex database service.
3Retrieval-Augmented Generation (RAG) Architecture
While large language models (LLMs) can generate human-like text, they often 'hallucinate' or produce answers not supported by specific source data, which is unacceptable for analytical tasks requiring evidence.
Implemented a RAG architecture: first, retrieve relevant complaint evidence from the vector store, then provide this evidence as context to the generation model (FLAN-T5).
Ensured that all generated answers were grounded in actual retrieved complaint excerpts, significantly enhancing the trustworthiness and verifiability of the system's output for analysts.
Engineering Challenges & Solutions
Real architectural bottlenecks encountered and the engineering rationale behind their resolution
#1Large Dataset Scale & Indexing Efficiency
Processing and indexing 1.37 million unstructured complaint records efficiently into a searchable format posed significant challenges in terms of computational resources and time.
Developed an optimized preprocessing pipeline for data cleaning and text chunking, followed by the creation of a persisted FAISS vector store. This allowed for scalable and efficient indexing.
#2Achieving Accurate Semantic Retrieval
Traditional keyword search proved inadequate for finding semantically related complaints, as different consumers often described similar issues using varied terminology.
Implemented sentence-transformer embeddings to capture the semantic meaning of complaint narratives, and used dense vector similarity search in FAISS to retrieve contextually relevant documents.
#3Ensuring Groundedness of Generated Answers
The risk of LLMs generating plausible but factually incorrect or unsupported answers was a major concern for an analytical tool requiring reliability and evidence.
Adopted a RAG approach where the LLM's (FLAN-T5) generation was strictly conditioned on specific, retrieved complaint excerpts. The UI also exposed this evidence.
Security, Integrity & Reliability
Production safeguards, boundary enforcement, and fault-tolerance patterns
Persisted FAISS Index
The FAISS vector index is persisted to disk, ensuring data durability and rapid reloading for continuous operation without re-indexing the entire dataset.
Deterministic Preprocessing
The data preprocessing pipeline is designed to be deterministic, ensuring consistent embeddings and retrieval results for identical input data.
Evaluation Pipeline
A dedicated evaluation framework and benchmark (CFPB complaints, business questions, retrieval rate metric) ensure the system's performance is measurable and auditable.
Evidence Visibility
The Gradio interface explicitly displays the raw complaint excerpts used to formulate answers, allowing human analysts to verify the grounding and build trust in the system.
Separation of Concerns
The RAG pipeline components (preprocessing, indexing, retrieval, generation, UI) are logically separated, enhancing modularity, maintainability, and testability.
Verified Results & Status
Factual metrics, operational milestones, and current production state
- An AI-powered RAG application successfully processing and indexing 1.37 million CFPB consumer complaints.
- Demonstrated 100% retrieval rate on the documented benchmark, where the top-5 relevant chunks were found for all benchmark queries.
- Implemented a comprehensive RAG pipeline showcasing data preprocessing, semantic embedding, vector indexing, retrieval, and grounded generation.
- Developed an evidence-based Gradio interface for interactive querying and transparent display of generated answers with supporting complaint excerpts.
- This project serves as a robust proof-of-concept for applying RAG to large-scale, unstructured datasets for analytical insights.
Lessons Learned & Retrospective
Key technical takeaways that inform future platform architecture decisions
Building large-scale RAG systems necessitates meticulous data preprocessing and chunking before any meaningful evaluation of retrieval quality can be performed.
Semantic retrieval, powered by dense embeddings, proves invaluable in datasets where users describe similar issues using diverse terminology, significantly enhancing discovery over keyword search.
Implementing evidence panels within the user interface is critical for building user trust and making generated answers inspectable and verifiable, addressing the 'black box' problem of LLMs.
It is essential to understand and evaluate retrieval quality and generation quality as distinct problems, as improvements in one do not automatically guarantee improvements in the other.
Christian Digital Content Platform - Backend API
Backend & Admin System
Enterprise AI Knowledge Platform (RAG)
AI & RAG Engineering
Related Case Studies
Swift Addis Car Detailing Platform
A production car detailing platform for appointment booking, service management, staff scheduling, availability, and business operations, backed by a Node.js/Express API and a React administrative dashboard.
Enterprise AI Knowledge Platform (RAG)
A high-throughput RAG system enabling real-time semantic document search across internal repositories with pgvector, BM25 hybrid ranking, and document-level RBAC.