How to Choose an AI Model for Document Extraction

Choose an AI model for document extraction by testing your twenty worst documents, not by reading benchmarks. The decision comes down to four things: whether your documents are clean digital PDFs or photographs of paper, whether layout carries meaning, how the vendor bills image tokens, and what your tolerance is for a wrong number reaching your database. Benchmarks answer none of those, because benchmark documents are cleaner than yours.

Here is the process that produces a defensible pick in about a day.

Start by classifying your documents, not the models

Extraction difficulty varies more across document types than across models. Sort a real sample into three buckets before you look at any vendor.

Digital-native text. PDFs generated by software, HTML invoices, exported statements. The text layer already exists. A vision model is often the wrong tool here because you can extract the text cheaply and hand plain text to a cheaper model. This is the bucket where people overspend.

Structured scans. Photographed or scanned forms with a consistent layout: purchase orders, lab results, claim forms. Layout carries meaning, so position matters. Vision models earn their price here.

Messy real-world captures. Phone photos of receipts, handwriting, rotated pages, fax artefacts, stamps over text. This bucket determines your model choice, because it is where the difference between models becomes large.

Most teams have all three and size the whole pipeline around the first bucket, then get surprised by the third.

The four questions that decide it

1. Does layout carry meaning?

If the answer to "which column is this number in" changes what the number means, you need a model that sees the page. A table where the same figure means charge or credit depending on position is a layout problem, and stripping text out of the PDF destroys the signal you need. If layout is decorative, text extraction plus a text model is cheaper and more reliable.

2. How are image tokens billed?

This is the cost variable people miss. Vendors do not price images uniformly. DeepSeek's V4-Flash-Vision-Exp release notes state that images are tokenized at the same rate as text and capped at 384 tokens per image, which makes per-page cost flat and predictable. Other vendors scale image tokens with resolution, which means a high-resolution scan of a dense page can cost several thousand tokens on its own.

Work out your cost per page, not your cost per million tokens. A 40-page contract at 1,500 image tokens per page is 60,000 input tokens before the model has produced a single character of output.

3. What happens when it is wrong?

There are two failure modes and they need different models. A model that leaves a field blank when unsure is easy to build around: route blanks to a human. A model that confidently invents a plausible invoice number is dangerous, because nothing downstream will catch it.

Test for this deliberately. Feed a document with a genuinely missing field and see whether you get null or fiction. A model that reliably says it cannot find something is worth more than a model that scores two points higher on average accuracy. Our post on making AI say it does not know covers the prompting side of getting that behaviour.

4. Do you need the same model for the whole pipeline?

Usually not. A common and cheap pattern is a small model doing classification and routing, then a stronger model only on the documents that need it. Classifying a page as "clean digital invoice" or "photo of a receipt" is a trivial task for a cheap model, and it lets you spend the expensive model's budget on the 15% of documents that actually need it.

Build the eval set from your worst twenty documents

This is the step that makes the difference, and it takes about two hours.

1. Pull 20 real documents, weighted toward the ugly end. Include at least one rotated page, one with handwriting, one with a stamp or watermark over text, one multi-page document where the answer is on page 7, and one where a field you expect is genuinely absent.

2. Hand-key the correct answers into a spreadsheet. Yes, by hand. This is your ground truth and it needs to be right.

3. Run every candidate model against all 20 with the same prompt.

4. Score three columns separately: exact-match correctness, hallucination count (a value returned that does not exist in the document), and refusal count (a field left blank that was present).

Score them separately because they trade off. The model with the highest correctness is frequently also the model with the most hallucinations, and for financial data that is the wrong trade.

Prompting matters more than the model on easy documents

On bucket one and bucket two documents, a well-specified prompt closes most of the gap between a mid-tier and a frontier model. Give the model the exact field names you want, the format for each, and explicit instructions on what to return when a field is absent. Ask for a fixed structure rather than prose. We covered the technique in prompting AI to extract data from a document, and getting the output shape stable is covered in getting JSON output from AI.

On bucket three documents, prompting helps much less. That bucket is where you find out whether the model can actually see.

Decide, then re-decide on a schedule

Model capability in this specific area has moved fast, and the model you picked six months ago may now be both worse and more expensive than a current mid-tier option. Put a calendar reminder to rerun your eval set quarterly. The set already exists and the run takes an hour, which makes it one of the highest-return recurring tasks in an extraction pipeline. The general method is in how to test a new AI model before switching, and the background on what these models are doing is in what is a vision language model.

For the broader question of matching model class to job across your whole product, see which AI model for which task. And to keep track of when a new option is worth evaluating at all, how to keep up with AI news is the sane version of following releases.

FAQ

Is a vision model better than traditional OCR for document extraction?

For messy or variable documents, usually yes, because a vision model reads meaning rather than characters. For high-volume, fixed-layout, clean scans, dedicated OCR is often faster and cheaper and its errors are more predictable. Many production pipelines use OCR first and a vision model only on the pages OCR flags as low confidence.

How many documents do I need to test a model properly?

Twenty carefully chosen documents beat two hundred random ones. What matters is coverage of failure modes, not sample size. Add a new document to the set every time production surprises you, and the set gets better over time.

Can I use a cheap model for document extraction?

On clean digital documents, yes, and you probably should. On photographs, handwriting, or dense multi-column layouts, cheap models fail in ways that are expensive to catch downstream. Route by document type rather than picking one model for everything.

How do I stop a model inventing values that are not in the document?

Instruct it explicitly to return a null value for anything it cannot locate, ask it to quote the source text next to each extracted value, and reject any answer whose quoted source does not appear in the document. That last check is mechanical and catches most fabrication.

Should I fine-tune a model for my document type?

Rarely worth it as a first move in 2026. Prompting plus routing plus a good eval set usually gets you to acceptable accuracy at a fraction of the effort. Consider fine-tuning only when you have a large labelled set, a stable document format, and a measured gap that prompting cannot close.