Need Architecture Review: Building a Low-Latency Local Hybrid-RAG Chatbot for a University Campus

AI Discussions
1

Hello DeepLearning.AI Community,

I am designing and developing an end-to-end, localized campus chatbot for my university. My primary goals are zero API/cloud costs, low answer latency, and a deep understanding of internal university documents combined with live campus web updates.

Since I will have to defend this system architecture in an upcoming technical review, I want to build the core pipeline using a clean, raw open-source stack without relying on heavy abstractions like LangChain or LlamaIndex. This ensures I understand every single layer of data flow.

Here is my current architectural plan:

1. Current Architecture Stack:

  • Data Ingestion & Chunking: Python scripts parsing raw student handbooks, PDFs, and course catalogues into fixed-size character chunks.
  • Vector Database: Local ChromaDB (persistent client running locally). [1, 2]
  • Embeddings Model: all-MiniLM-L6-v2 via Chroma’s default pipeline. [1]
  • LLM Core Engine: Ollama running a quantized 4-bit model (e.g., Llama 3.1 8B or Phi-3) to optimize tokens-per-second and maintain low memory usage. [1, 2, 3, 4]
  • Orchestration: Pure Python script that computes query embeddings, runs cosine similarity via ChromaDB, handles basic prompt engineering, and streams token responses back. [1]

2. The Requirement (The “Hybrid” Aspect):

The bot needs to address two distinct data paths:

  1. Static Internal Queries: Looking up institutional facts (hostel fees, library timings, office locations) stored in the vector database.
  2. Dynamic Live Queries: Pulling fresh online notices from the active university website or live search results for real-time announcements.

3. Where I Need Your Help & Expertise:

I want to ensure my data engineering and pipeline choices are mathematically sound and production-ready. I would appreciate your insights on the following:

  1. Query Routing Logic: What is the industry standard for routing a user query between the local Vector DB and a live Web Search API? Should I use a lightweight “router LLM” to classify the intent first, or trigger both searches concurrently and let the final system prompt handle context filtering?
  2. Chunking and Retrieval Strategy: Given that university handbooks contain highly specific tables (e.g., fee structures, grading metrics), standard recursive character chunking might split relevant rows. What strategies do you recommend for handling mixed text and tabular data inside ChromaDB without complex cloud services?
  3. Local Latency Optimization: To keep generation fast on standard consumer hardware, I am relying on 4-bit quantization. Beyond streaming tokens, are there any hidden performance bottlenecks within local embeddings or database querying that I should watch out for?

I want to truly learn the mechanics of this stack rather than just copying code templates. Any advice, reading resources, or architectural critiques would be incredibly valuable!

Thank you in advance,
Sai Pradyumna Thiriveedi