How are you handling academic data collection for research-based AI projects?

AI Discussions
, ,
1

Finished a project last month where I needed a retrieval system grounded in recent ML and biomedical literature. Sharing what I learned on the data side since most resources focus on the retrieval architecture and skip over the messy part.

The biggest practical issue was PDF extraction quality. Raw PyMuPDF output on two-column journal layouts was consistently breaking paragraph flow and merging footnotes into body text. Switched to pulling pre-extracted full text directly from ScholarAPI instead of handling PDFs myself. Coverage is open-access only but for ML, CS, and biomedical literature that covers the majority of what you actually need.

The part that made the biggest difference for retrieval quality was not the chunk size or embedding model.. it was having clean coherent input text to begin with.

Happy to share the ingestion code if useful. Curious whether others here have hit the same PDF parsing issues and how you handled it.

4

Two-column PDF layouts, footers, and floating figures are major bottlenecks in RAG pipelines. PyMuPDF reads top-to-bottom across the whole page by default, which merges columns.

If you want to keep parsing PDFs locally rather than relying on external APIs, here are the most effective alternatives:

  1. Meta’s Nougat: A vision-based transformer trained specifically on scientific papers. It processes page images and outputs clean Markdown with LaTeX math syntax intact, natively solving the two-column problem.
  2. Grobid: The academic standard for parsing scientific literature. It uses machine learning to extract metadata, sections, and structured text into TEI-XML format.
  3. LlamaParse: A cloud-based parser designed specifically for RAG. It maintains document layout, tables, and multi-column formatting extremely well.
  4. Unstructured.io: An open-source partitioning library. It offers layout-detection models that segment the document into structural elements (titles, text, tables) before extracting text.
  5. PyMuPDF Coordinate Sorting: If you must stick to PyMuPDF, you can sort text blocks by their bounding box coordinates. Sorting blocks by x (horizontal) coordinate first, and then y (vertical) coordinate, helps group columns before joining the text.