flowchart TB
A[Document image] --> B[VLM]
B --> C[Structured output]
4 Architecture Patterns
NoteExecutive summary
- Three reference architectures dominate 2026 agentic OCR: single-pass VLM (Pattern A), agentic loop with reflection (Pattern B), and hybrid CV+VLM (Pattern C). Each is valid for a different intersection of document variability and stakes.
- The choice is not “which architecture is best.” The choice is “which architecture matches my workload.” Pattern A is right for low-stakes, low-variability work; Pattern B is the 2026 default for general agentic workloads; Pattern C is winning the high-stakes benchmarks and the high-stakes verticals (healthcare, legal, finance) by a non-trivial margin.
- Tool chains are not optional. A reflection loop with no specialized tools is a model talking to itself. The vendors leading 2026 benchmarks all route work to specialized extractors at the region level — that is what is winning.
- HITL gates are not a fallback bolted on the side. In production they are part of the architecture, and the rate of escalation (1–10% in healthy deployments) is one of the most honest signals you can read about a system.
4.1 A working architecture, walked through
Before the abstractions, a concrete system. Anterior AI runs healthcare prior-authorization workflows in production (Reducto AI 2026). Their reported pipeline, simplified for the purpose of this walk-through:
A clinical packet arrives — a referral letter, an ICD-coded diagnosis sheet, a stack of supporting lab reports and prior office notes, often a fax cover page with a phone number that has been scribbled over and corrected. Total length, ten to forty pages. The packet enters a layout-first analysis: a CV model partitions every page into regions, classifies each region (header, prose paragraph, table, signature block, form field, handwritten margin note, image), and emits a region map. A router dispatches each region to the right extractor — printed text regions go through one path, hand-written notes through another, tables through a specialized table extractor, signature blocks to a stamp-and-signature classifier. The extracted regions assemble back into a structured document object whose schema is the prior-auth schema, not a generic markdown blob.
A reflection step then validates the assembled object against schema rules: dates are in expected ranges, ICD codes resolve, the patient ID matches across pages, the requested procedure is on the payer’s covered-procedures list. Anything that fails validation triggers either a re-extraction (with a revised prompt that names the specific issue) or, if re-extraction has burned its retry budget, an escalation to a human reviewer with the failing field highlighted on the source page.
The system reports about 99.24% field-level accuracy on its golden set. The HITL escalation rate sits in the low single digits. The architecture has all three of the things the rest of this chapter is going to abstract: a layout-first decomposition, a router that sends regions to specialized tools, a reflection loop with both retry and escalation paths. It is, in 2026 vocabulary, a Pattern C architecture.
This chapter unpacks the three architectures, gives you criteria for choosing between them, and is honest about the engineering each one costs.
4.2 Pattern A: single-pass VLM
The simplest architecture in modern agentic OCR is the one that, by the definition in Chapter 3, barely qualifies as “agentic” at all.
A document image goes in. A vision-language model emits structured output — typically JSON conforming to a schema declared in the prompt or via the model’s structured-output API. There is one model call. There is no reflection, no validation loop, no specialized tool, no retry, no grounding by default. Latency is around 200ms to 2s per page depending on model and document complexity. Cost is the cost of one VLM call.
This architecture works, and it works well, on a specific kind of workload: documents whose layout is simple, whose tables are small or absent, whose fonts are clean, whose stakes for any single error are low, and whose schemas are stable. A consumer-facing app that lets a user photograph a receipt and pulls out total, merchant, and date is a textbook Pattern A workload. So is bulk indexing of email attachments where 95% accuracy is good enough because the only downstream consumer is a search box.
Pattern A’s failure modes are not edge cases; they are central tendencies of the workloads it gets misapplied to:
- Tables. A single-pass VLM in 2026 will still routinely merge adjacent table cells, drop rows when the table crosses a page break, and confuse rowspan/colspan in tables with merged headers. This is not a model-quality problem that another year of training will fix; it is a structural limitation of single-pass extraction. The Reducto vs. hyperscaler gap on RD-TableBench — about twenty percentage points of table similarity in Reducto’s favor — is the most reliable signal of this fact (Reducto AI 2025).
- Long documents. Even with current context windows, the practical accuracy of single-pass extraction degrades over multi-page documents. The model attends to early pages well and forgets the middle. This shows up in production as fields that come from page 1 being reliably extracted and fields that come from page 8 being silently dropped.
- Grounding. Pattern A emits structured output with no built-in connection to source regions. You can ask the model to include bounding-box coordinates and it will, but the coordinates are themselves model outputs — they are not ground truth, and they are wrong some non-trivial fraction of the time. Pattern A is the architecture that produces beautiful JSON that you cannot audit.
- Confidence. A single-pass VLM produces output with no calibrated probability that the output is right. Anything that claims to be a confidence score in Pattern A is, at best, a model’s opinion about its own work.
Pattern A is not bad. It is not “agentic OCR” in the strict five-attribute sense — it satisfies, at best, two of the five attributes — but it is the right architecture for workloads where the cost of an error is low and the document variability is modest. The mistake is using it for a workload it cannot serve and then being surprised when production accuracy falls short of the demo.
The 2026 representative implementations of Pattern A are the hyperscaler APIs (Google Document AI, AWS Textract, Azure Document Intelligence) running in their default modes, frontier VLM APIs called with a structured-output prompt (GPT-4V/4o, Claude, Gemini, Qwen3-VL), and the cheapest tier of specialist vendors (LlamaParse Fast, Mistral OCR 3 Standard).
4.3 Pattern B: agentic loop with reflection
Pattern B is the 2026 default for general-purpose agentic OCR. It is what the field talks about when it says “agentic.” It satisfies all five attributes in the Chapter 3 scorecard, and it does so without requiring the engineering investment that Pattern C demands.
flowchart TB
A[Document] --> B[Plan]
B --> C[Extract pass]
C --> D{Confidence ok?}
D -- No --> E[Reflect / re-prompt]
E --> C
D -- Yes --> F[Validate]
F --> G{Schema ok?}
G -- No --> E
G -- Yes --> H[Output + provenance]
The shape is: plan, extract, critique, re-prompt if necessary, validate, output with provenance. Each step has a real engineering parameter behind it.
Plan. The system articulates, before extracting, what it is looking for. In simpler implementations this is the schema injected into the prompt. In more sophisticated implementations the system generates a per-document extraction plan that varies by document type — different field priorities, different validation rules, different retry strategies. The plan step is what makes the architecture goal-driven in the Chapter 3 sense.
Extract pass. A VLM call (or sequence of calls for long documents) produces a candidate extraction. This is structurally similar to Pattern A but happens inside a loop that knows the extraction might not be final.
Confidence check. This is where Pattern B’s engineering quality varies most between implementations. The naïve version compares a model-emitted confidence score against a threshold. The sophisticated version cross-checks the candidate extraction against external oracles — schema validators (does it parse? do dates lie in plausible ranges?), tool outputs (does the layout detector agree on where this field is?), secondary models (does a second VLM with a different prompt produce the same answer?). The difference between naïve and sophisticated confidence checks is, in practice, the difference between detection failures and detection successes.
Reflect / re-prompt. If the confidence check fails, the system re-prompts with a revised instruction. The revision typically references the specific failure — “the date you extracted as 2026-13-45 is not valid; please re-examine the date field” — rather than rerunning the same prompt. This is why retry-thrash is a real failure mode: a system that re-prompts identically each time learns nothing and converges on nothing.
Validate. A final structural check against the target schema. Schema validation is cheap and surprisingly diagnostic. Many extraction errors that confidence checks miss show up as schema failures (a required field is null; a numeric field contains “TBD”; an enum field contains an unexpected value).
Output with provenance. Every field in the final output carries a pointer to the source region. This is what makes Pattern B grounded in the Chapter 3 sense.
The retry budget — typically two to four attempts before escalation — is the most visible engineering parameter, and the one most worth asking vendors about. Too low and the system gives up on documents it could have handled. Too high and a single recalcitrant document can run up cost and latency without ever converging.
Pattern B is well-suited to general-purpose agentic OCR: messy enterprise documents, mixed layouts, moderate stakes. Most LlamaParse Agentic (LlamaIndex and Kaggle 2026) and LandingAI ADE (Landing AI 2026) deployments are Pattern B. The strengths are flexibility (one architecture handles a wide variety of document types) and engineering tractability (the loop is conceptually simple even if the implementation is not). The cost is, well, cost — multi-pass means multiple VLM calls, which means real money per page, plus the latency penalty of running through a loop.
Pattern B’s failure modes are subtler than Pattern A’s. The architecture detects more errors but can produce silent hallucinations when the reflection step approves a wrong answer (Chapter 5 unpacks this in detail). It can also produce retry-thrash on documents that the model is genuinely incapable of extracting correctly — burning the retry budget without improving the answer. Production deployments need observability into both failure modes, and most teams discover this fact the hard way.
4.4 Pattern C: hybrid CV+VLM
Pattern C is the architecture that is winning the high-stakes benchmarks and the high-stakes verticals. It is also the architecture that costs the most engineering to build and operate.
flowchart TB
A[Document] --> B[Layout detector / CV]
B --> C[Region routing]
C --> D1[Text blocks → OCR]
C --> D2[Tables → table extractor]
C --> D3[Figures → VLM]
D1 --> E[Assembler]
D2 --> E
D3 --> E
E --> F[Validator]
F --> G[Output]
The core move is layout-first decomposition. Before any extraction happens, a computer-vision model partitions the page into regions and classifies each one. The classification is fine-grained enough to drive routing: text blocks go to a dedicated text-extraction path, tables go to a specialized table extractor, figures go to a VLM, signature blocks go to a signature classifier, handwritten regions go to a handwriting-tuned extractor.
The reason this architecture wins on adversarial workloads is that specialization beats generality where the distribution is narrow and the stakes are high. A table extractor that has been trained specifically on tables, with table-specific evaluation metrics (Needleman-Wunsch cell alignment, header-disambiguation accuracy), is better at tables than a general VLM that has been trained on everything including tables. The Reducto result on RD-TableBench — about 0.90 cell-alignment similarity on adversarial tables, roughly twenty points ahead of the hyperscaler APIs in Pattern A — is the clearest data point (Reducto AI 2025). PP-StructureV3 making the same point at under 100 million parameters, matching Gemini-2.5-Pro on OmniDocBench at a tiny fraction of the compute cost, is the strongest argument that this is structural rather than incidental (PaddlePaddle 2025).
Pattern C also has the strongest grounding story by construction. Because the system tracks regions from the layout step onward, every extracted field carries a region pointer all the way through to the output. There is no “model emitted a bounding box and we hope it’s right” — the bounding box came from the layout detector, not the extractor.
The cost of Pattern C is the engineering. Every component is a real piece of software with its own training data, eval, and failure modes. The layout detector needs to be accurate or every downstream extractor receives garbage. The router needs to know which extractor to send each region to, which means the classification taxonomy needs to be maintained as document types evolve. The assembler needs to handle cases where regions overlap, where the layout detector segments wrongly, where extractors disagree on overlapping regions. Each of these is solvable. None of them is free.
Pattern C is what’s behind Reducto’s hybrid architecture, IBM Granite-Docling and the Docling pipeline framework, OpenDataLab’s MinerU, and the PaddleOCR / PP-Structure lineage. It is also implicit in any in-house implementation that takes high-stakes extraction seriously enough to refuse the “one model does everything” path.
It is worth naming that the hyperscaler ecosystems also expose layout-detector primitives that can serve the same architectural role inside a Pattern C pipeline, for buyers already committed to a cloud vendor:
- Azure AI Document Intelligence Layout ($10/1k pages) — text, tables, paragraphs, section headings, selection marks, figure regions with bounding boxes. The Azure-native answer for the layout-extraction stage.
- Google Document AI Layout Parser ($10/1k pages) — Gemini-powered layout extraction plus initial chunking in a single call; closest hyperscaler equivalent to LlamaParse Agentic for buyers on GCP.
- AWS Textract AnalyzeDocument with Layout feature ($10–$25/1k pages, stackable with Forms/Tables/Queries) — newer addition to the Textract feature set; less mature than Azure or Google Layout but functional.
These hyperscaler Layout APIs serve the same architectural role as PP-StructureV3 or Granite-Docling in the layout-detector slot of Pattern C — they emit structured regions that downstream extractors can route off. A pipeline that uses Azure DI Layout to segment, an LLM (frontier or Granite-Docling) to extract from text blocks, and a specialized table extractor for table regions, plus a schema validator and HITL queue, is a valid Pattern C architecture for organizations committed to Azure. The same shape works on GCP with Document AI Layout Parser and on AWS with Textract AnalyzeDocument Layout. Chapter 7 walks the hyperscaler sub-products in more detail.
The honest framing: Pattern C is the architecture for workloads where errors cost more than engineering does. Healthcare prior-authorization fits because a denied claim costs hundreds to thousands of dollars in downstream rework, dwarfing the marginal engineering cost of a hybrid pipeline. Legal contract review fits because a missed clause can be a six-figure legal exposure. KYC document review fits because a wrongly approved high-risk account can be a regulatory event. Commodity invoice processing rarely fits, because the marginal error cost is too low to justify the engineering.
4.5 Document classification — the often-missed primitive
The three reference architectures above describe the extraction core. In production deployments handling more than one document type — which is most production deployments — there is a step that conceptually precedes extraction and meaningfully changes the architecture: document classification.
The classification step’s job is to decide what kind of document this is before deciding how to extract from it. An incoming PDF could be an invoice, a contract, a claim form, a clinical note, a driver’s license, or a fax cover sheet with the actual document on the back. The downstream extraction logic depends on the answer: an invoice goes through a vendor-specific schema; a contract goes through clause extraction; a claim form goes through a different pipeline with HITL gating tuned to claim severity.
The architectural shape is classify-then-route:
flowchart TB
A[Document] --> B[Classifier]
B --> C{Document type}
C -- Invoice --> D1[Invoice pipeline]
C -- Contract --> D2[Contract pipeline]
C -- Claim --> D3[Claim pipeline]
C -- Unknown --> D4[HITL triage]
D1 --> E[Structured output]
D2 --> E
D3 --> E
D4 --> E
This is a Pattern C variation (sometimes integrated into a larger Pattern C; sometimes a stand-alone routing layer in front of Pattern B pipelines). Three common implementations in 2026:
- Lightweight classifier model. A small, fast specialized classifier — typically a fine-tuned BERT-class model or an embedding-plus-kNN approach — produces a document-type label in milliseconds. Cheap, accurate on in-distribution types, fails ungracefully on novel types. The right default for narrow-but-stable document distributions where the class set is small and known.
- LLM-based zero-shot classification. A VLM or LLM call with a prompt like “Classify this document as one of: invoice, contract, claim, clinical note, other.” Slower and more expensive per document than a dedicated classifier, but flexible — new document types can be added by editing the prompt rather than retraining a model. The right choice when the document distribution is wide and evolving, or when the classification needs to consider visual cues (a logo, a layout fingerprint).
- Layout-based heuristics. Cheap rule-based classifiers using bounding-box patterns from the layout-detector stage. (“If there are exactly four large text blocks in the top quadrant and a table below, it’s an invoice.”) Brittle but free; useful as a confidence-boosting cross-check for the LLM-based classifier rather than as the sole classifier.
The hyperscaler equivalents are first-class. AWS Bedrock Data Automation and Azure DI Prebuilt models effectively do classification + extraction in one call by routing to a specific blueprint or prebuilt model; Databricks’ ai_classify is the explicit primitive. The hyperscaler-platform tier has converged on classify-then-route as the default architecture, which is one indicator that it is the right default.
4.5.1 Why classification gets missed
The single largest 2026 procurement mistake involving classification is assuming the parser also handles it. Most parsers — including the agentic-OCR specialists — do not classify. They extract whatever they are pointed at against whatever schema they are given. If you point an invoice parser at a contract, you get an extraction that conforms to the invoice schema and is mostly nonsense. The system did its job; the system was given the wrong job.
The fix is to put classification in front of extraction as an explicit step, with its own evaluation, its own monitoring, and its own HITL escalation path for novel or low-confidence cases. Teams that build agentic OCR pipelines without classification routinely encounter a class of production failures that look like “the extractor is wrong” but are actually “the extractor was given a document outside its design distribution and had no way to flag the mismatch.”
4.5.2 Failure modes when classification is wrong
A wrong classification fails worse than a missing extraction, because the system is now confidently doing the wrong thing. The two failure modes:
- Misclassification followed by confident extraction. The system classifies a contract as an invoice, routes to the invoice pipeline, and emits a structured “invoice” with fields scraped from the contract. The values are plausible (vendor name, dates, dollar amounts all exist in contracts) but wrong. Downstream consumers may not notice until reconciliation fails weeks later.
- Novel-class blindness. A document arrives that does not match any known class. A well-engineered classifier returns “unknown” with low confidence and routes to HITL triage. A poorly-engineered classifier returns its closest-match class with moderate confidence, sending the novel document into the wrong extraction pipeline. The defense is an explicit “unknown / out-of-distribution” class in the classifier’s label set, and routing on that class to HITL by default.
Both failures are catchable with the eval discipline of Chapter 11 — if classification is measured as a separate accuracy axis from extraction. Bundling them into a single “did we get the answer right” metric hides exactly the failures that originate in classification.
4.6 Tool chains, memory, and HITL gates
The three patterns above describe the extraction core. Production agentic OCR systems have additional components that the simplified diagrams elide.
Tool inventory. A production system typically has access to: a layout detector (Pattern C central; optional for Pattern B), a table extractor, a schema validator (always), an external knowledge-base lookup for entity resolution (vendor IDs, ICD codes, ISO country codes), a redaction module for PII, a classification model for document-type routing, and a fallback OCR for when the primary extractor fails. Tool-using in the Chapter 3 sense means these are routinely invoked, not just technically available.
Memory. Short-term memory carries context within a single document — heading hierarchy, units of measurement, vendor identity established on page 1 — into extractions on later pages. Long-term memory carries vendor-specific corrections from HITL queues back into prompts, document-type priors learned across deployments, and audit logs of model behavior for drift detection. Few public vendors talk much about their memory architectures, but the gap between vendors that have solid memory engineering and those that do not is sharply visible in production over a six-month window.
HITL gates. Every production architecture has a path to a human reviewer. The question is where in the pipeline the gate sits. The healthier patterns place HITL gates post-validation, before output, with the failing field highlighted on the source page and the candidate extraction pre-populated for the reviewer to accept, reject, or correct. The unhealthier patterns dump entire documents into a queue with no annotation, requiring the reviewer to find the issue manually. The healthy version cuts reviewer time per case from minutes to seconds.
The escalation rate is one of the most diagnostic production signals. A healthy 2026 agentic OCR deployment escalates somewhere between 1% and 10% of documents. Below 1% suggests the system is silently swallowing errors that should have escalated; above 10% suggests the extraction is too unreliable for the workload (or that the eval rules are too strict, but that’s the easier failure to diagnose).
4.7 Choosing between the three patterns
The decision reduces to two axes: document variability and stakes.
- Narrow variability, low stakes → Pattern A. Don’t pay for a loop you don’t need.
- Wide variability, low or moderate stakes → Pattern B. The 2026 sweet spot for most enterprise agentic OCR.
- High stakes, any variability → Pattern C with HITL gates. Engineering cost pays for itself fast when the alternative is downstream rework cost.
Volume modifies the decision but does not change its shape. At very low volume (under 10k pages/month), Pattern A via a hyperscaler API is almost always the right choice regardless of stakes — the math does not support investing in Pattern B or C until volume justifies it. At very high volume (millions of pages/month), the open-source Pattern B or C path (Granite-Docling, PaddleOCR + Pattern C orchestration, OlmOCR-2 self-hosted) starts to pay for itself versus commercial APIs.
A practical heuristic: build the simplest pattern that satisfies your error budget, not the most sophisticated pattern your engineering can support. Sophistication has a maintenance cost. The vendors that look impressive at demo time but become operational burdens six months in are almost always the ones who deployed Pattern C when Pattern B would have done the job.
The chapters that follow assume these three patterns as vocabulary. When Chapter 7 ranks commercial vendors, the ranking implicitly is “which pattern do they implement well, and at what price.” When Chapter 11 talks about evaluation, the rubric varies by pattern (the metrics that matter for Pattern A are not all the same as the metrics that matter for Pattern C). When Chapter 13 enumerates anti-patterns, several of them are specific ways each of the three patterns fails when misapplied.
TipNamed takes
If your architecture has no place for a table-specific tool, you’re betting the model is uniformly good at everything. Twelve months of leaderboards say it isn’t.
Hybrid CV+VLM is winning the table benchmarks for the same reason ensembles win on tabular ML: specialization beats generality where the distribution is narrow and the stakes are high.
Build the simplest pattern that satisfies your error budget. Sophistication you do not need is technical debt with a long fuse.