Multimodal RAG

RAG beyond text: indexing images, tables, and diagrams from documents. PDF processing, multi-vector retrieval, and using vision models for table and image understanding.

Quick Reference

→Real documents contain text, tables, images, and diagrams — all carry information
→PDF processing: extract text, tables, and images as separate elements with different embedding strategies
→Multi-vector retrieval: store text summaries of visual elements as separate vectors alongside raw content
→Vision models (GPT-5.4, Claude) can describe images and tables, creating text representations for embedding
→Unstructured.io is the best library for parsing complex documents into structured elements

Beyond Text: The Multimodal Challenge

Most RAG tutorials assume documents are pure text. Real-world documents are multimodal: financial reports have charts and tables, technical docs have architecture diagrams, product manuals have screenshots, and research papers have figures. If you only index the text, you miss critical information stored in these visual elements. A table comparing product features might be the most important part of a document, but a text-only RAG pipeline ignores it entirely.

▸Tables: comparison tables, pricing tables, data tables — structured information that's often the answer to a query
▸Images: architecture diagrams, screenshots, charts, graphs — visual information that text can't fully capture
▸Code blocks: syntax-highlighted code in documents — needs special handling to preserve formatting
▸Equations: mathematical formulas in research papers — LaTeX or rendered images
▸Embedded documents: PDFs with embedded spreadsheets, slides with embedded charts

Two strategies for multimodal content

Strategy 1 (Summarize): Use a vision model to describe images and tables as text, then embed the text description. Simpler, works with any vector store. Strategy 2 (Multi-vector): Embed visual elements with a multimodal embedding model (like CLIP) alongside text embeddings. More complex but preserves visual information better.

Extracting Elements from PDFs

PDF processing is the most common multimodal RAG challenge because PDFs are the dominant document format in enterprises. The key is to extract text, tables, and images as separate elements, each with appropriate metadata. The Unstructured library is the best open-source tool for this — it uses a combination of OCR, layout analysis, and table detection to parse complex PDFs.

Multi-Vector Retrieval Strategy

The multi-vector approach stores two things for each element: (1) a text summary for embedding and retrieval, and (2) the raw content (full text, HTML table, image path) for inclusion in the LLM context. This decouples what you search on (optimized text summaries) from what you show the LLM (the full original content). This is especially powerful for tables and images — you search using a natural language summary but pass the full table or image to the LLM.

Sign in to read this article

This is a premium article. Sign in with your Google account to continue.