Vision LLMs for Structured Extraction: Turning Invoices and Forms into Validated Data

Most teams that try to extract data from images follow the same path. They send a photo of an invoice to a vision model, ask for JSON, parse the result, and write it to the database. It works on the first ten documents. On document eleven, a coffee stain covers the total, the model guesses a number, and nothing in the pipeline notices.

The fix is not a better prompt. It is three decisions made up front: force the output into a schema you control, validate it with code the model cannot talk its way past, and give the pipeline an explicit way to say "I don't know" that routes the document to a person.

This post builds that pipeline in Python. It uses the Anthropic SDK, but the shape carries over to any provider with image input and tool calling.

The problem with "return JSON"

Asking for JSON in free text has three failure modes:

1. Syntax errors. The model wraps the JSON in prose or markdown fences, or emits a trailing comma.
2. Schema drift. It renames total_amount to total on one document and drops currency on another.
3. Silent guessing. The most dangerous one. The model fills a field it could not read with a plausible value, and the output looks perfectly valid.

Native structured output features and tool-use schemas fix the first two. The third needs your own logic, because a schema can require a field to be present but cannot know whether the model actually saw it. That is why the design below gives the model an escape hatch: every extracted field carries a confidence level, and "unreadable" is a legal value.

Step 1: Define the schema with an escape hatch

Use Pydantic to describe what you want. Each field is wrapped so the model must say how sure it is.

from datetime import date
from enum import Enum
from typing import Generic, Optional, TypeVar

from pydantic import BaseModel, Field

T = TypeVar("T")


class Confidence(str, Enum):
    high = "high"        # clearly legible, unambiguous
    low = "low"          # legible but partially obscured or ambiguous
    unreadable = "unreadable"  # not visible; value must be null


class Field_(BaseModel, Generic[T]):
    value: Optional[T] = None
    confidence: Confidence


class LineItem(BaseModel):
    description: Field_[str]
    quantity: Field_[float]
    unit_price: Field_[float]


class Invoice(BaseModel):
    vendor_name: Field_[str]
    invoice_number: Field_[str]
    invoice_date: Field_[date]
    currency: Field_[str] = Field(description="ISO 4217 code, e.g. INR, USD")
    total: Field_[float]
    line_items: list[LineItem]

Two details matter here. The unreadable level plus a nullable value means the model has a valid way to decline a field without breaking the schema. And the wrapper makes confidence a first-class output you can act on, not something you hope the model reports correctly in prose.

Step 2: Call the model with a forced tool

Pass the schema as a tool and force the model to call it. This removes the prose wrapper entirely.

import base64
import json
from pathlib import Path

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

MODEL = "claude-sonnet-5-5"  # check the official model list before deploying

EXTRACT_TOOL = {
    "name": "record_invoice",
    "description": (
        "Record the fields read from this invoice image. "
        "If a field is not visible, set value to null and confidence to 'unreadable'. "
        "Never infer a value that is not printed on the document."
    ),
    "input_schema": Invoice.model_json_schema(),
}


def extract_invoice(image_path: Path, media_type: str = "image/jpeg") -> dict:
    image_b64 = base64.standard_b64encode(image_path.read_bytes()).decode()

    response = client.messages.create(
        model=MODEL,
        max_tokens=2048,
        tools=[EXTRACT_TOOL],
        tool_choice={"type": "tool", "name": "record_invoice"},
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "image",
                        "source": {
                            "type": "base64",
                            "media_type": media_type,
                            "data": image_b64,
                        },
                    },
                    {
                        "type": "text",
                        "text": "Extract the invoice fields from this image.",
                    },
                ],
            }
        ],
    )

    tool_block = next(b for b in response.content if b.type == "tool_use")
    return tool_block.input

The tool_choice forces the call, so there is always a tool_use block to read. The schema comes from Pydantic, so your validator and the tool definition cannot drift apart.

Step 3: Validate with code, not with the model

Do not trust the returned dict. Validate it against the same Pydantic model, then run business rules the schema cannot express.

from pydantic import ValidationError


def validate_invoice(raw: dict) -> tuple[Invoice | None, list[str]]:
    problems: list[str] = []
    try:
        invoice = Invoice.model_validate(raw)
    except ValidationError as exc:
        return None, [f"schema: {err['loc']} {err['msg']}" for err in exc.errors()]

    # Arithmetic check: a total that does not reconcile is the strongest
    # signal that the model misread a digit somewhere.
    computed = sum(
        (li.quantity.value or 0) * (li.unit_price.value or 0)
        for li in invoice.line_items
    )
    declared = invoice.total.value or 0
    if declared and abs(computed - declared) > max(1.0, 0.01 * declared):
        problems.append(
            f"total mismatch: declared {declared}, line items sum to {computed:.2f}"
        )

    # Any field the model marked unreadable is a routing decision, not an error.
    for name in ("vendor_name", "invoice_number", "invoice_date", "total"):
        if getattr(invoice, name).confidence == Confidence.unreadable:
            problems.append(f"unreadable: {name}")

    return invoice, problems

The arithmetic check is the most valuable line in the whole pipeline. Line items times quantities should reconcile with the printed total. When they do not, something was misread, and no amount of prompt tuning will tell you which digit. Flagging the mismatch is cheap; trusting it is expensive.

Step 4: Route by outcome, not by hope

Three outcomes, three destinations:

from dataclasses import dataclass


@dataclass
class Routed:
    status: str          # "auto_accept" | "human_review" | "reject"
    invoice: Invoice | None
    problems: list[str]


def route(image_path: Path) -> Routed:
    raw = extract_invoice(image_path)
    invoice, problems = validate_invoice(raw)

    if invoice is None:
        return Routed("reject", None, problems)  # schema failed: retry or reject

    if problems or any(
        f.confidence != Confidence.high
        for f in (invoice.vendor_name, invoice.invoice_number, invoice.total)
    ):
        return Routed("human_review", invoice, problems)

    return Routed("auto_accept", invoice, problems)

Notice what does not happen. A document with a low-confidence total never reaches the database automatically, even if the arithmetic happens to pass. Confidence and arithmetic are two independent checks, and both must agree before you trust a value.

Step 5: Measure it on a labelled set

None of this is useful if you cannot tell whether it is getting better. Build a small labelled set before you tune anything:

• Collect 100 to 200 real documents, including the bad ones: skewed phone photos, faded thermal receipts, handwriting on forms.
• Label the true values by hand, once, in a spreadsheet or JSON file.
• Track three numbers per release: field-level accuracy on fields the pipeline auto-accepted, auto-accept rate, and false accept rate (wrong values that passed without review).

The third number is the one that matters. A pipeline that auto-accepts 95% of documents with a 2% false accept rate is worse than one that auto-accepts 70% with 0.1%, because the errors are invisible. Set your threshold by looking at the false accept rate first and the throughput second.

Comparison: where each piece earns its place

Layer	What it catches	What it misses
Free-text JSON prompt	Nothing reliably	Almost everything
Forced tool schema	Syntax errors, missing keys, wrong types	Plausible wrong values
Pydantic validation	Type and shape violations, bad dates	Misreads that still type-check
Arithmetic reconciliation	Digit misreads in totals and line items	Errors that cancel out, vendor name typos
Confidence routing	Fields the model admits it cannot read	Confident mistakes
Human review queue	Everything the above flags	Nothing, but it costs people time

No single row is enough. The pipeline works because the layers catch different failure modes.

Common pitfalls

Letting the model infer missing values. A prompt that says "extract the total" will produce a total even when none is printed. Say explicitly to return null, and check the output for values that do not appear in the image.

Trusting confidence labels without calibration. The model's "high" is not a probability. Calibrate it against your labelled set: if fields marked high are wrong 5% of the time, that is your real error rate and your review threshold should reflect it.

Ignoring image quality. Downscaling a 4000-pixel scan to fit a size limit can erase the decimal point on a total. Check the resolution you send against what the provider documents, and keep the original file for the review queue so humans see what the model saw.

Mixing currencies and locales silently. 1,234.50 and 1.234,50 both appear in real invoices. Store the currency as a separate validated field, and do not let the model normalise separators without you checking the result against the arithmetic check.

Skipping the review loop. The review queue is where your labelled set grows. Every human correction is a new test case. Feed corrections back into the golden set monthly, or the pipeline will drift from the documents you actually receive.

Next steps

1. Pick one document type you handle today, not every type at once. Invoices or receipts are a good start because they have arithmetic to check.
2. Build the labelled set first, even if it is only 50 documents, and record your current manual process's error rate as the baseline.
3. Implement the schema, the forced tool call, and the arithmetic check. Run the pipeline on the labelled set and read every false accept by hand.
4. Only then tune the confidence threshold and the prompt.

The model is the least reliable part of this system and the easiest to replace. The schema, the arithmetic, and the labelled set are what keep the output trustworthy when you swap it out.

---

Building AI systems that have to be right, not just fluent, is what our cohorts focus on: evals, guardrails, and production pipelines. If you want to work through this kind of pipeline with engineers doing the same, see the AI Engineering cohort.