Hi community,
I’m currently mapping out an ingestion pipeline for a research-focused RAG system. While setting up the vector storage and embedding models is straightforward, building a stable data layer to fetch academic papers and citation metadata is turning out to be the real challenge.
Initially, I looked into building custom scrapers to extract data directly from public scholarly engines. However, dealing with constant IP blocks, CAPTCHAs, and cleaning messy HTML format is incredibly resource-intensive and unreliable for production.
To keep up with model inference, it feels like an API-first infrastructure is necessary. I’ve been looking into structured alternatives like ScholarAPI to bypass the whole proxy/scraping management headache and pipe clean JSON metadata directly into the workflow.
I wanted to ask the experts here:
-
For those working with scientific or medical LLM applications, what does your ingestion stack look like?
-
Do you prefer processing raw PDFs offline using parser libraries, or have you integrated automated APIs to feed clean data into your pipelines?
Would love to know what strategies or tools you’ve found to be the most resilient!