My Journey Building a Universal LLM Data Pipeline — Current Challenges

AI Discussions
1

I’m working on a project to build a universal ingestion and reasoning pipeline for LLMs. The vision is simple: I want my assistant to take any file — Excel, CSV, JSON, PDF, DOCX, TXT — and answer structured queries.

But as I’ve been building this, I’ve run into several pending problems that I think other engineers, professors, or practitioners might have faced too.

:small_blue_diamond: The Tools and Models I’m Using

  • Chainlit — for building the conversational interface.

  • FAISS — for semantic search and indexing.

  • Postgres — for structured persistence.

  • SentenceTransformers — for embeddings and similarity search.

  • Qwen3 — as the reasoning LLM backbone.

  • PyTorch — for model operations.

  • OpenPyXL and CSV DictReader — for Excel and CSV ingestion.

  • PyPDF2 + pytesseract — for PDF text and OCR fallback.

:small_blue_diamond: The Problems I’m Facing

  • Import confusion: My project has its own utils.py, but libraries like sentence-transformers also have a util.py. At times, Python imports the wrong one, leading to errors like “cannot import name 'sanitize_row’”.

  • Duplicate definitions: Functions like sanitize_row, chunk_rows, and ingest_file ended up defined in multiple files (utils.py, file_ingestion.py, qwen3_chat_interface_multiturn.py). This caused mismatches and confusion.

  • Persistence mismatch: I had two versions of persist_file_content — one expecting a single argument (file_path), another expecting four (file_name, headers, rows, text_content). Mixing them caused runtime crashes.

  • Excel/CSV ingestion: This is my main pain point. Queries like “highest progress of 100%” aren’t consistently answered. Sometimes the model speculates instead of filtering properly. Analysts need structured, reproducible results.

:small_blue_diamond: What I’m Trying to Achieve

  • A single source of truth for ingestion helpers inside utils.py.

  • A unified persistence pipeline that chunks structured and unstructured data correctly, indexes it in FAISS, and stores it in Postgres.

  • A universal query interpreter that can detect relevant columns (even with fuzzy names like “Progress” vs “Completion”), filter rows, and return results in a clean table or JSON.

  • Professional responses that are structured, reproducible, and explainable — something a data analyst would trust.

:small_blue_diamond: Why I’m Sharing This

I believe many AI engineers and professors working on LLM‑driven data systems face similar challenges. My hope is that by sharing my current struggles, I can learn from others who have solved these problems — or at least hear different perspectives on how to approach them.

:small_blue_diamond: Call for Opinions

  • How do you handle import conflicts between project files and library modules?

  • What’s your best practice for avoiding duplicate function definitions across files?

  • How do you design a universal ingestion pipeline that works across multiple file types, especially Excel and CSV?

  • And most importantly: how do you ensure LLM responses are structured and professional enough for data analysts?

:herb: In plain words: I’m building something ambitious, but I’m stuck on these technical and design challenges. I’d love to hear from others — your opinions, advice, or even critiques. Maybe your insights can help me upgrade my system to the “5/5” professional standard I’m aiming for.