Automating Document Processing With AI: Invoices, Contracts, and Forms

How to Use AI to Extract Data from Invoices and Documents

The fastest path to reliable AI document extraction is a four-stage pipeline: OCR to get clean text, a structured LLM extraction pass to produce a JSON object, schema validation to reject malformed output, and a human-review queue for any field that falls below your confidence threshold. That sequence, run end-to-end with measurements at every stage, is what separates a demo that works on clean PDFs from a system that handles 10,000 invoices a month in production.

I am Mahmoud Zalt, an independent senior AI systems architect with 16 years of production software experience. I have designed and shipped AI automation pipelines for document-heavy workflows including invoices, contracts, onboarding forms, and insurance claims. You can read more about my background, browse past projects, or go straight to the AI Automation service page to see how I engage with teams on exactly this kind of work.

What Teams Get Wrong Before They Write a Single Line of Code

Most teams start by calling an LLM with 'extract all fields from this invoice' and then declare success when it works on three sample documents. The failure mode shows up at scale: edge cases like rotated scans, handwritten totals, multi-currency line items, or invoices from vendors who use non-standard layouts. The three root mistakes I see repeatedly are:

• No baseline accuracy measurement. You cannot improve what you do not track. Run your pipeline on a labeled held-out set of 200 to 500 real documents before deploying anything.
• Single-pass extraction. Asking one LLM call to 'do everything' produces inconsistent JSON. Separating extraction from validation catches errors before they hit your database.
• Treating the LLM as infallible. A model that hallucinates a total of $10,000 instead of $1,000 is worse than no automation at all. You need guardrails, not just prompts.

The Four-Stage Extraction Pipeline

Stage 1: OCR and Document Normalization

For digital-native PDFs (computer-generated, not scanned), use a PDF parsing library such as pdfplumber or pymupdf to extract raw text with positional metadata. For scanned documents or images, use a dedicated OCR service: AWS Textract, Google Document AI, or Azure Form Recognizer all support table detection, which matters for invoice line items. Do not use a general-purpose vision model as your OCR layer. Purpose-built OCR engines return bounding boxes and confidence scores per word. You will use those scores later.

Normalize the output into a single canonical format before passing it downstream: plain text with section markers, or a structured key-value block if the OCR service supports that. Strip headers, footers, and page numbers unless they contain data.

Stage 2: Structured LLM Extraction

Pass the normalized text to an LLM with a strict prompt that asks for a JSON object matching a known schema. Use function calling or structured output mode (available in GPT-4o, Claude 3.x, and Gemini 1.5) so the model is constrained to valid JSON. Never ask for free-form prose and then try to parse it. A minimal prompt for an invoice looks like this:

Extract the following fields from the invoice text below.
Return ONLY valid JSON matching this schema.
Schema: { vendor_name: string, invoice_number: string,
invoice_date: ISO8601, due_date: ISO8601,
line_items: [{description: string, quantity: number,
unit_price: number, total: number}],
subtotal: number, tax: number, total_due: number,
currency: ISO4217 }
If a field is not present, return null. Do not invent values.

The 'do not invent values' instruction matters. Models will fill gaps with plausible-looking fiction if you do not explicitly forbid it.

Stage 3: Schema Validation and Business Rule Checks

Validate the returned JSON against a strict schema using Pydantic (Python), Zod (TypeScript), or your language's equivalent. Schema validation catches type errors. But schema-valid output can still be logically wrong. Add business rule checks on top:

• Sum of line_items[].total should equal subtotal within a rounding tolerance of +-0.02.
• total_due should equal subtotal + tax.
• invoice_date should be before due_date.
• Currency codes must be in a known allowlist.

Any document that fails a business rule check goes to the human review queue regardless of confidence score. A mathematically impossible invoice is not a low-confidence extraction. It is a failed extraction.

Stage 4: Confidence Scoring and Human-in-the-Loop Review

Assign a confidence score to each extracted field. Use a combination of: the OCR word-level confidence scores for fields whose value can be traced back to specific tokens, LLM self-reported confidence (ask the model to rate each field 0-1 in a parallel call), and cross-validation signals (did the math check out, did the vendor name match a known vendor list). Route any field below your threshold (I typically start at 0.85 and tune from data) to a lightweight review UI where a human confirms or corrects the value. Log every human correction. Those corrections are your retraining signal.

Worked Example: Invoice Processing at 2,000 Documents per Month

A B2B SaaS client was processing vendor invoices manually across a 4-person finance team. The goal was to cut review time by 80% without increasing error rate. Here is what the pipeline looked like in production:

End result: 94% of invoices processed automatically with zero human touch. The remaining 6% went to the review queue, down from 100% manual before. Review time per document dropped from 8 minutes to under 90 seconds because reviewers only touched flagged fields, not the whole document. Total LLM cost was roughly $0.012 per invoice at GPT-4o pricing with caching on the system prompt.

Contracts and Forms: Where the Pattern Changes

Invoices are structured. Contracts and long-form agreements are semi-structured at best. The extraction pipeline is the same, but the prompt strategy and chunking change significantly.

Contracts

Long contracts exceed context windows if you try to extract everything in one pass. Instead, chunk the document by section (use a semantic splitter or just split on heading patterns), extract fields per chunk, then merge and deduplicate. For contracts, the fields you usually care about are: parties, effective date, termination date, governing law, limitation of liability clause, auto-renewal terms, and payment terms. Each of those maps to a specific clause type. You can use a classifier to first identify which chunks contain relevant clauses, then run a targeted extraction prompt only on those chunks. This cuts cost by 60 to 80% versus extracting from the entire document.

Forms and Applications

Structured forms (PDFs with labelled fields) are the easiest case. AWS Textract and Google Document AI both have form extraction modes that return key-value pairs directly. You may not need an LLM at all for a clean, templated form. Use an LLM only when: the form is inconsistently formatted across submissions, fields use non-standard labels, or you need to interpret free-text answer fields. Use the simplest tool that achieves your accuracy target.

Treating Accuracy as a Measured Number, Not a Vibe

The most important discipline in document extraction is running evals before and after every change. An eval is a labeled dataset of documents where you know the correct output, paired with an automated comparison that scores field-level accuracy. Build this from day one, not as an afterthought.

The metrics I track per pipeline version:

• Field-level accuracy: for each field type (total_due, vendor_name, etc.), what percentage of extractions match the ground truth exactly or within tolerance.
• Human-review rate: what percentage of documents hit the review queue. This is your efficiency metric.
• Correction rate: of documents that went to review, how often did a human actually change a value. If the correction rate is below 5%, your confidence threshold is too low and you are sending unnecessary work to humans. If it is above 30%, your threshold is too high and you are auto-approving too many errors.
• Dollar error rate: for financial documents, track total dollar value of incorrectly extracted amounts as a percentage of total volume processed. This is the number your CFO cares about.

Run your eval suite on every prompt change. A prompt that improves vendor_name accuracy by 2% but degrades total_due accuracy by 1% is a net negative if total_due errors have higher downstream cost.

Security, PII, and Compliance Considerations

Invoices and contracts contain sensitive financial data, PII, and sometimes trade secrets. Before sending documents to any external LLM API, answer three questions:

1. Is there a DPA (Data Processing Agreement) in place with the model provider? OpenAI, Anthropic, and Google all offer enterprise agreements with DPAs. Do not use consumer-tier APIs for production financial documents.
2. Does your data residency requirement permit sending data to a US-based API? EU clients under GDPR may require that documents never leave the EU. Azure OpenAI and Google Cloud Vertex AI support EU regions. Self-hosted open-weight models (Mistral, Llama) are an option if cloud APIs are off the table.
3. Do you need to log document content for audit? If yes, encrypt at rest with customer-managed keys and implement access logging. If no, configure your pipeline to not persist raw extracted text beyond the processing window.

Also: redact PII from your eval dataset before storing it in source control. I have seen teams commit labeled invoice datasets with real vendor banking details to GitHub. That is a breach waiting for a disclosure deadline.

Going Further: Tool Calling, MCP, and Agentic Extraction

For simple extraction, a single LLM call with structured output is sufficient. When the task gets more complex, such as 'extract the invoice and then look up the vendor in our ERP system and flag if the total exceeds the PO amount', you are in agentic territory. Use a framework that supports tool calling: LangChain, LlamaIndex, or a minimal custom loop with the Anthropic or OpenAI tool-calling APIs. The Model Context Protocol (MCP) is worth evaluating for teams that want a standardized way to connect the extraction pipeline to internal systems like ERPs, CRMs, or approval workflows without custom per-integration code.

Keep the agentic layer thin. An agent that extracts data, looks up a vendor, and routes to an approval workflow is three tool calls. Do not build a six-agent orchestration system for a three-step process. I say this from watching multiple teams over-architect document pipelines by 6 to 12 months and then ship something slower and less reliable than a direct API integration would have been.

Frequently Asked Questions

Can I use ChatGPT or Claude directly to extract invoice data without building a pipeline?

For low volume (under 50 documents a month), yes. Upload the PDF, ask for the fields you need, and copy the output. For anything higher volume or feeding a database, you need a pipeline with validation and error handling. Manual copy-paste from a chat UI does not scale and has no audit trail.

How accurate is AI invoice extraction compared to a human?

A well-tuned pipeline on clean digital PDFs reaches 96 to 99% field-level accuracy, which matches or exceeds human data entry for volume work. On low-quality scans or handwritten documents, accuracy drops to 85 to 92% depending on OCR quality. The key is measuring your actual accuracy on your actual documents, not trusting vendor benchmarks run on clean benchmark datasets.

Which is better for document extraction: GPT-4o, Claude, or a fine-tuned model?

For standard structured extraction on English-language documents, GPT-4o and Claude 3.5/3.7 Sonnet are close in accuracy and cost. GPT-4o has an edge on structured output reliability. Fine-tuned models (fine-tuned GPT-3.5 or a self-hosted Mistral) beat both on cost at high volume once you have enough labeled training data (typically 500 to 2,000 examples). Start with a frontier model, collect corrections from your human review queue, and fine-tune once you have the data to justify it.

How do I handle invoices in multiple languages or formats?

Frontier models handle most European languages well. For date parsing, always normalize to ISO8601 in your schema prompt and explicitly tell the model the expected date format for the document locale. For currencies, always extract the currency code separately from the amount and validate against an allowlist. The most common failure mode in multilingual extraction is date format ambiguity: 04/05/2024 means April 5 in the US and May 4 in most of Europe. Make the model state which it is using.

What does it cost to process invoices with AI at scale?

At GPT-4o pricing with prompt caching, a typical invoice extraction prompt (2,000 to 3,000 tokens in, 500 tokens out) costs $0.008 to $0.015 per document. At 10,000 invoices a month, that is $80 to $150/month in LLM costs plus OCR costs ($0.005 to $0.015 per page for Textract or Document AI). Total pipeline cost at that volume is typically $200 to $400/month, well below the cost of a single hour of manual data entry labor.

Do I need to fine-tune a model for my specific invoice templates?

Usually no. Prompt engineering with a few-shot schema plus your business rules handles 90% of cases. Fine-tune only when you have a high volume of a specific document type (over 5,000 examples), accuracy on that type is measurably below your target after prompt tuning, and the cost savings from switching to a smaller fine-tuned model justify the fine-tuning investment. Most teams that jump to fine-tuning skip the prompt engineering step and leave significant accuracy gains on the table.

Ready to Build a Document Extraction Pipeline That Actually Works in Production?

If you are looking at a backlog of invoices, contracts, or forms that your team is still processing manually, the pipeline described in this article is buildable in weeks, not months. The bottleneck is rarely the AI. It is the OCR normalization, the validation rules specific to your document types, and the human-review workflow tuned to your team's process. Those are engineering and systems design problems, and they are exactly what I work on with clients.

Browse the AI Automation service page for details on how I engage, or go to the contact page to start a conversation about your specific document workflow. I work as an independent architect, which means you get direct access without the overhead of an agency engagement.