Document Parsing for RAG: A Practical Guide to Docling, LlamaParse, and Marker

Most RAG failures blamed on "the model" or "the embeddings" are actually parsing failures. If your ingestion step flattens a two-column research paper into interleaved gibberish, merges a table into one run-on sentence, or silently drops the text trapped in a scanned page, no amount of reranking or prompt tuning will save the answer. Retrieval quality is bounded by chunk quality, and chunk quality is bounded by how well you extracted structure from the source document.

This guide covers how to parse real-world documents — PDFs, DOCX, PPTX, scanned images — into clean, structure-aware Markdown, which tools to use, and the pitfalls that quietly degrade answer quality.

The problem: PDFs are a display format, not a data format

A PDF stores instructions for placing glyphs at coordinates on a page. It does not reliably store "this is a heading," "these cells belong to the same row," or "this paragraph continues on the next page." When you run a naive text extractor (pdfminer, PyPDF2, pdftotext), you get the glyphs back in roughly the order they were drawn, which is often not reading order.

Concrete failure modes:

• Multi-column layouts get read left-to-right across both columns, splicing unrelated sentences together.
• Tables collapse into a wall of numbers with no row/column association.
• Headers, footers, and page numbers get injected into the middle of paragraphs every ~500 tokens.
• Scanned pages contain zero extractable text — you need OCR.
• Reading order for figures, captions, and sidebars is unpredictable.

Each of these produces chunks that embed poorly (mixed topics), retrieve poorly (diluted signal), and confuse the LLM at generation time (broken tables, orphaned numbers).

What "good parsing" produces

The target output for RAG is structure-aware Markdown (or a structured JSON document you can serialize):

• Headings as #/##/### so you can chunk on semantic boundaries and attach section paths as metadata.
• Tables as Markdown tables (or HTML) so cell relationships survive.
• Reading order corrected for columns and page breaks.
• Headers/footers/page furniture stripped.
• Figures and captions kept together, ideally with a text description.
• Page numbers preserved as metadata for citations, not inline noise.

Markdown is the pragmatic intermediate representation: LLMs understand it natively, it is cheap to chunk on structure, and every serious parser now emits it.

The tool landscape in 2026

Tool	Type	Layout / tables	OCR	Best for	Notes
Docling (IBM / Linux Foundation)	Open source, self-host	Strong (layout model + table structure model)	Yes (Tesseract, EasyOCR, RapidOCR)	Default choice for on-prem / private data	Ships granite-docling compact vision model; built-in chunkers
LlamaParse	Hosted API	Very strong (vision-model mode)	Yes	Complex financial/scientific docs, when you can send data out	Usage-priced per page; "premium"/vision tiers cost more
Marker	Open source, self-host	Strong, fast	Yes	High-throughput PDF-to-Markdown batches	GPU strongly recommended for speed
Unstructured	Open source + hosted	Moderate to strong	Yes	Many file types, existing LangChain pipelines	"hi_res" strategy needed for good layout; slower
MarkItDown (Microsoft)	Open source	Weak on scanned/complex PDF	Limited	Office files, quick conversions	Great for DOCX/PPTX/XLSX, not a heavy PDF engine
Cloud OCR (AWS Textract, Google Document AI, Azure Document Intelligence)	Hosted API	Strong on forms/tables	Yes	Forms, invoices, regulated workflows with SLAs	Priced per page; strong table/KV extraction

Rules of thumb:

• Private or regulated data that cannot leave your infra → Docling or Marker, self-hosted.
• Complex layouts and you can use an API → LlamaParse vision mode or a cloud Document AI service.
• Mostly Office documents → MarkItDown or Docling.
• You already run LangChain/LlamaIndex ingestion → Unstructured or the framework's Docling integration.

Code: a Docling ingestion pipeline

Install:

pip install docling docling-core

Convert a document to Markdown:

from docling.document_converter import DocumentConverter

converter = DocumentConverter()
result = converter.convert("https://arxiv.org/pdf/2408.09869")  # URL or local path
doc = result.document

markdown = doc.export_to_markdown()
print(markdown[:2000])

DocumentConverter auto-detects the format (PDF, DOCX, PPTX, HTML, images, and more). For scanned PDFs, enable OCR explicitly:

from docling.document_converter import DocumentConverter, PdfFormatOption
from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import PdfPipelineOptions

pipeline_options = PdfPipelineOptions()
pipeline_options.do_ocr = True
pipeline_options.do_table_structure = True
pipeline_options.table_structure_options.do_cell_matching = True

converter = DocumentConverter(
    format_options={
        InputFormat.PDF: PdfFormatOption(pipeline_options=pipeline_options)
    }
)

API surfaces move fast. Check the current Docling docs for the exact option names before you ship — treat the snippet above as the shape, not gospel.

Chunk on structure, not character count

The reason to parse into structure is to chunk on it. Docling ships a HybridChunker that respects document hierarchy and merges small fragments up to a token budget:

from docling.chunking import HybridChunker
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("BAAI/bge-small-en-v1.5")

chunker = HybridChunker(tokenizer=tokenizer, max_tokens=512, merge_peers=True)

chunks = list(chunker.chunk(doc))

for c in chunks[:3]:
    # contextualize() prepends the heading path, e.g. "Methods > Data Collection"
    enriched = chunker.contextualize(chunk=c)
    print(enriched[:300])
    print("---")

The contextualize() call is doing real work: it prepends the section path to each chunk so an isolated paragraph about "sample size" still carries "Methods > Participants" into the embedding. That single trick measurably improves retrieval on long structured documents.

Wire it into a vector store

import chromadb
from chromadb.utils import embedding_functions

client = chromadb.PersistentClient(path="./chroma")
ef = embedding_functions.SentenceTransformerEmbeddingFunction(
    model_name="BAAI/bge-small-en-v1.5"
)
collection = client.get_or_create_collection("docs", embedding_function=ef)

for i, c in enumerate(chunks):
    meta = c.meta.export_json_dict() if hasattr(c.meta, "export_json_dict") else {}
    collection.add(
        ids=[f"doc-{i}"],
        documents=[chunker.contextualize(chunk=c)],
        metadatas=[{
            "headings": " > ".join(meta.get("headings", []) or []),
            "page": (meta.get("doc_items", [{}])[0].get("prov", [{}])[0].get("page_no")
                     if meta.get("doc_items") else None),
        }],
    )

Keep page and headings in metadata — you will want them for citations and for filtered retrieval later.

Code: LlamaParse for the hard documents

When layouts defeat local tools (dense financial tables, nested cells, chart-heavy decks), a vision-model API often wins:

from llama_parse import LlamaParse

parser = LlamaParse(
    api_key="llx-...",
    result_type="markdown",
    parse_mode="parse_page_with_agent",  # vision mode; higher cost per page
    high_res_ocr=True,
)

documents = parser.load_data("annual_report.pdf")
print(documents[0].text[:2000])

Cost discipline: run a cheap local parser first, measure table/answer quality on a sample, and only route the documents that fail to the paid vision tier. Parsing every page at premium rates gets expensive fast at corpus scale.

Evaluate your parser before you trust it

Do not eyeball three pages and call it done. Build a tiny harness:

1. Pick 15–25 representative documents — include your worst offenders (scans, multi-column, table-heavy).
2. Write 5–10 questions per document whose answers live in tables, footnotes, or section 2 of a two-column layout.
3. Run the full pipeline (parse → chunk → retrieve → generate) per parser.
4. Score: answer correctness (LLM-as-judge or manual), retrieval hit rate (did the right chunk come back?), and table fidelity (spot-check rendered Markdown tables against the source).
5. Track cost and latency per page alongside quality.

Often a free self-hosted parser is within a few points of a paid API on your corpus — and the gap only matters on a small, identifiable subset you can route separately.

Common pitfalls

Pitfall	Symptom	Fix
Naive text extraction on complex PDFs	Interleaved sentences, broken tables	Use a layout-aware parser (Docling/Marker/LlamaParse)
Skipping OCR on scanned pages	Chunks silently missing; "not in context" answers	Detect image-only pages, force OCR
Fixed-size character chunking after good parsing	Tables split mid-row, headings orphaned	Chunk on structure with a token-aware hybrid chunker
Dropping page/section metadata	No citations, no filtered retrieval	Persist page and headings per chunk
Header/footer text left inline	Repeated noise every ~500 tokens pollutes embeddings	Enable furniture removal / post-filter repeated lines
Parsing the whole corpus at premium API rates	Surprise bill	Cheap parser first, route only failures to vision tier
No parser evaluation	Quality regressions ship unnoticed	Maintain a 20-doc Q&A eval set, re-run on any change
Ignoring images/charts entirely	Questions about figures always fail	Generate captions/descriptions for figures, index them as text

Checklist before shipping ingestion

• Layout-aware parser chosen and justified against data-residency constraints
• OCR path tested on a genuinely scanned document
• Output is structure-aware Markdown, not flat text
• Chunking respects headings and a token budget; small fragments merged
• Each chunk carries page + heading path metadata
• Headers, footers, page numbers stripped from chunk bodies
• Tables spot-checked against source for cell fidelity
• 20-document Q&A eval set exists and passes at your target threshold
• Per-page cost and latency measured; expensive parsing routed, not blanket-applied
• Re-ingestion is idempotent (stable chunk IDs, safe to re-run)

Next steps

1. Take five of your worst documents and run them through Docling today. Diff the Markdown against your current extractor's output.
2. Build the 20-document Q&A eval set. It is the single highest-leverage artifact in a RAG project and it takes an afternoon.
3. Add page and headings to chunk metadata if they are not there yet, and expose page-level citations in your UI.
4. Only after parsing and chunking are solid, revisit embeddings and reranking — that is where tuning actually pays off once the inputs are clean.

---

Want to go deeper on RAG systems — parsing, chunking, evals, and agentic retrieval — with hands-on review of your pipeline? That is exactly what we build in the Joinloop AI Engineering cohort.

Sources: Fastio, LlamaIndex, Firecrawl, Docling docs, LangChain Docling integration