Internal Chatbot

RAG powered chatbot to streamline technical support and knowledge retrieval.

RoleFull-Stack Dev

Year2025

Links

RESTRICTED

Overview & Problem Statement

The Support division handles a high volume of user inquiries through a ticketing system. Previously, both new and experienced employees had to manually search through a large number of resolved tickets, making the process time consuming. To address this, I developed a chatbot that acts as an internal knowledge retrieval system. Instead of manually searching, users can ask questions and receive relevant past ticket resolutions instantly. The system also integrates additional application level technical metadata to provide better context when resolving issues. By centralizing these resources, the chatbot reduces the time needed to search for past solutions and helps the team resolve issues more efficiently.

My Role & Contributions

End-to-End RAG Architecture

Designed and built the full Retrieval Augmented Generation (RAG) pipeline to generate context aware responses based on internal data.

Vector Database Setup

Integrated a vector database to enable fast and relevant semantic search across historical tickets and metadata.

Data Engineering & Curation

Processed and structured raw data, filtering only the necessary information to keep the knowledge base relevant and usable.

Tech Stack & Tools

Ruby on RailsLangChainLocal LLMAPI-based LLMVector DatabaseMySQL

Key Features / Highlights

Natural Language Search

Users can ask troubleshooting questions in plain language without relying on rigid keyword filters.

Query Routing

Implements intent based routing to determine whether to retrieve past ticket resolutions or application related information.

Source Citations

Each response includes references to original data sources to reduce hallucinations and improve trust.

Challenges & Learnings

Handling Tabular Data in RAG

Challenge:

LLMs struggled to understand raw tabular data from Excel.

Solution:

Built a custom parser to transform tables into a structured narrative format based on how domain experts interpret the data.

Result:

Improved data usability and made tabular information retrievable within the RAG pipeline.

Optimizing Retrieval Accuracy

Challenge:

Retrieving relevant results for technical queries and past ticket resolutions was inconsistent.

Solution:

Implemented hybrid search combining embeddings and keyword based retrieval, and benchmarked multiple embedding models.

Result:

Achieved more accurate and stable retrieval for both keyword based and natural language queries.

LLM Selection & Prompt Engineering

Challenge:

Needed a fast and reliable LLM that could follow strict guidelines and avoid hallucinations.

Solution:

Evaluated several models and designed system prompts with guardrails to ensure responses stayed grounded in retrieved data.

Result:

Improved response consistency, reduced hallucinations, and maintained a consistent tone.

Outcome & Results

From Research to Implementation

Initially started as a research experiment and later approved for full development after being presented in an internal team meeting.

Internal Deployment

Developed into a functional application and deployed in the internal staging environment.

Used by Support Team

The application is now used by the Support team to find relevant ticket resolutions more efficiently, reducing manual search effort and helping resolve issues faster.