In a pharma R&D engagement, we watched a text-only RAG pipeline confidently answer a dosing question while the actual evidence, a dose-response curve and a table of adverse events, sat on the same page as image pixels the retriever never saw. The system was not wrong by accident. It was wrong by design, because the ingestion step had already discarded the highest-value content before a single embedding was computed.
In scientific and technical documents, the load-bearing claims live in figures, charts, and tables. A PDF-to-text extractor flattens those into nothing, or worse, into garbled fragments of axis labels. Your retriever then ranks prose that merely refers to a figure, and your model answers from the caption instead of the data. For regulated research, that is not a quality gap. It is an audit failure waiting to surface.
Why text-only ingestion silently fails
Most pipelines run a single text extraction pass and treat anything non-textual as noise. Three failure modes follow. Tables lose their row and column structure, so a value detaches from its header and its units. Charts vanish entirely, taking the trend or threshold with them. Multi-column and multi-figure layouts get linearized into the wrong reading order, fusing unrelated paragraphs into one chunk. None of these throw an error. The pipeline reports full coverage while half the meaning is gone.
Extract with a vision model, not an OCR pass
The fix is to treat each page as an image and let a vision-capable model read it. In our pharma build we ran page renders through GPT-4 Vision and GPT-4o to produce a structured representation: figures described and the underlying values transcribed, tables emitted as Markdown or JSON with headers intact, and chart trends stated explicitly so they become retrievable text. OCR alone does not get you here. OCR returns characters. A vision model returns the relationship between a curve, its axes, and the claim the figure supports, which is what a researcher actually queries against.
Keep the rendered page image and its bounding boxes alongside the extracted text. You will need them for citation, and they let a reviewer confirm the model transcribed a number correctly rather than hallucinated it. Do not throw the pixels away a second time.
Chunk on structure, embed for both modalities
Structure-preserving chunking is the difference between a retriever that finds the evidence and one that finds the sentence next to it. A figure and its caption are one unit. A table and its header row are one unit. A results section stays whole rather than being split mid-argument by a fixed token window.
A workable decision rule: never let a chunk boundary cut through a table or separate a figure from its caption; split on document structure first, then fall back to token limits only inside a section. Embed the vision-extracted descriptions as text so charts and tables become first-class retrieval targets, and use text-embedding-3-large with a vector store such as Pinecone. Carry the page number, figure ID, and bounding box as metadata on every chunk.
Page and figure citations are the product
For regulated research, an answer without a verifiable source is unusable. Every generated claim should resolve to a page number and, where it came from a figure or table, the specific figure ID and the cropped region. We render the cited crop back to the reviewer so they can confirm the model read the right cell of the right table. This turns the system from a plausible-text generator into a reference-cited tool a scientist will actually trust, because trust here means being able to check the source in seconds.
What it buys the business
In the pharma R&D engagement, literature reviews that took researchers weeks dropped to hours, and the answers came back with citations down to the figure. The reason it worked was not a larger model. It was refusing to discard the charts, figures, and tables at ingestion, the exact content the questions were really about. If your RAG system answers from prose while the evidence sits in the figures, you are not retrieving your documents. You are retrieving their captions.
