Working on a project where the outputs need to be grounded in actual published research, not just what the model already knows.
The challenge I kept running into was getting clean full text from papers programmatically. Abstracts are not enough for serious grounding. You need the actual content, the methods, the findings, the nuance that lives in the body of the paper.
After trying a few approaches including PDF parsing libraries and various academic APIs, I landed on using ScholarAPI which indexes open access papers and returns pre-extracted full text directly. No parsing headaches. The /list endpoint with indexed_after also lets you pull newly published papers automatically which was important for keeping the knowledge base current.
But I am genuinely curious how others here are handling this. Specifically:
When you are building RAG systems or grounded AI applications that need to reason over scientific literature, where does your source content actually come from?
Are you working with raw PDFs and cleaning them yourself, using existing academic databases, or something else entirely?
And for those who have tried multiple approaches, what made the biggest practical difference in output quality, the retrieval method, the chunking strategy, or the quality of the source documents going in?
Would love to hear what is actually working for people building in this space.