11 Evaluating Your Own Pipeline

Executive summary

The single most important investment in agentic OCR is a golden set of 200–500 hand-labeled documents drawn from your actual production traffic. Build it before you choose a vendor. Teams that skip this step pick the wrong vendor with predictable regularity.
Metric selection is domain-dependent. Character Error Rate is wrong for structured extraction. Field-level F1 is the default. Table-cell accuracy needs alignment (Needleman-Wunsch), not Jaccard. Schema-validity rate is cheap and surprisingly diagnostic. Pick the metric that predicts your downstream cost.
The HITL queue is part of your eval, not a fallback bolted on the side. Reviewer accept/correct rates are the most honest production-accuracy signal you will get, and most teams underutilize them.
The number that predicts your monthly bill is cost-per-correct-extraction-with-provenance. It is computable from three measured numbers (API cost, accuracy, HITL escalation rate × HITL cost). It is the answer; the rest is mostly noise.

11.1 Why the public benchmarks won’t save you

A common 2026 procurement story: a team picks the highest-scoring vendor on ParseBench, deploys to production, and watches accuracy fall by 5–10 percentage points on Day 1. The benchmark score was real. The production gap is real. The difference is that ParseBench’s distribution and the team’s production distribution are not the same.

This is not a problem with ParseBench. The benchmark is well-designed for what it does, and what it does — measure end-to-end agentic-parsing quality on a curated set of high-stakes enterprise documents — is genuinely useful. The mistake is treating any public benchmark as a forecast of your production behavior. No public benchmark, however well-designed, predicts your performance on documents the benchmark has never seen.

The discipline that closes the gap is straightforward to describe and operationally rare to execute. Build a golden set. Measure the right metrics. Run regression tests. Treat HITL as part of the eval. Compute the cost-per-correct-extraction number. Repeat quarterly.

This chapter walks each of those steps. Most of it is not new — every team operating agentic OCR in production has run into these requirements eventually. The hope of this chapter is that you hit them deliberately rather than after the third production incident that the public benchmarks did not predict.

11.2 Golden sets — the only thing that matters

A golden set is a collection of hand-labeled documents from your production traffic, against which every parsing decision is evaluated. Its purpose is to give you a measurement that does predict your production behavior, because it is your production behavior — sampled, labeled, and made into a stable evaluation target.

The discipline has four parts:

Selection. Pull 200–500 documents from your actual production traffic. Stratify by document type, source vendor, and difficulty. Include the edge cases your team complains about most loudly — the ones that the system gets wrong consistently — not the easy cases your current pipeline already handles. The temptation to over-represent clean documents because they are faster to label is the most common golden-set construction error.

Labeling. Hand-label each document with the schema your downstream consumers actually use. Field by field. Not the schema in a vendor’s documentation — your schema, the one your downstream system expects. Label the cases the system finds ambiguous: missing fields, illegible regions, contested values. The labels embed the team’s judgment about correctness, and that judgment is the load-bearing artifact.

Provenance metadata. For each labeled field, capture the source region (page number, bounding box) where the value originated. This makes the golden set usable not just for accuracy measurement but for grounding accuracy measurement — does the vendor’s reported source region match the actual region. Grounding measurement is a separate axis, and the golden set is the only place you can measure it reliably.

Maintenance. Add 50 new documents per quarter, drawn from recent production. Retire documents that have become unrepresentative (vendors that no longer exist, document types that no longer appear). The golden set is a living artifact, not a one-time project. Teams that label once and never update produce evaluations that predict behavior on the 2024 distribution while their production distribution moved on.

The labor cost is real. A 200-document golden set with thorough labeling and provenance metadata typically takes one to three weeks of focused effort by a labeler who understands the domain. The labor cost is also small relative to the cost of picking the wrong vendor. Teams that resist building a golden set on cost grounds routinely spend ten times the labor cost on a wrong-vendor rollback six months later.

A specific recommendation worth highlighting: label the golden set with the same people who will be operating the HITL queue. Their judgment about what constitutes a correct extraction is the judgment that will govern the production system. Labels from a separate “evaluation team” drift away from operational reality and produce eval scores that do not match what the HITL reviewers find in practice.

11.3 Metric selection per domain

The wrong metric quietly destroys vendor selection. The choice of metric needs to match the downstream cost of an error, and the right choice varies by workload type.

Character Error Rate (CER). The metric that classic OCR has used for thirty years. Correct for plain-text OCR — does the system get the characters right. Wrong for structured extraction — penalizes formatting differences that do not matter (extra whitespace, equivalent date formats, capitalization variants on enum values) and under-penalizes structural errors (a wrong field value in a correctly-extracted character stream).

CER is the right metric for: scanning books, indexing email attachments, accessibility-focused OCR. It is the wrong metric for most agentic-OCR workloads.

Field-level F1. The default metric for structured extraction. Per-field precision (of the fields the system extracted, what fraction match the golden label) and recall (of the fields in the golden label, what fraction did the system extract). F1 is the harmonic mean.

Field-level F1 is the right metric for: invoices, forms, contracts, claims, KYC documents, any workload where the output is a fixed-schema JSON object. The right granularity is per-field, not aggregate: per-field F1 tells you which specific fields the system handles well and which it struggles with, which is operationally diagnostic. Aggregate F1 averages across fields and hides the variance.

Table-cell accuracy with alignment. Tables need their own metric. Jaccard similarity on the set of cell values is wrong because it does not respect cell position — a system that gets every cell value right but in the wrong row and column scores perfectly on Jaccard and is, in fact, useless.

The right alignment metrics are Needleman-Wunsch (used in RD-TableBench (Reducto AI 2025)) or Hungarian assignment, both of which match predicted cells to ground-truth cells with positional constraints and penalize misalignments. The exact choice between Needleman-Wunsch and Hungarian depends on whether row/column order matters more (Hungarian) or whether sequence alignment is the right model (Needleman-Wunsch). For most enterprise tables, Hungarian is the safer default.

Schema-validity rate. Cheap, deterministic, and surprisingly diagnostic. Does the system’s output parse as your downstream schema? Are required fields present? Are dates in valid ranges? Are enums populated with allowed values?

Many extraction errors that confidence checks miss show up as schema failures. A system with 95% field-level F1 and 80% schema-validity rate is producing per-field correct outputs that aggregate into structurally invalid documents — a pattern you want to catch in eval, not in production.

Grounding accuracy. For compliance workloads, does the vendor’s reported source region for a field actually contain the extracted value? Measured as the IoU (intersection over union) between predicted and ground-truth bounding boxes, or as a binary “does the predicted region contain the value” check.

Without grounding accuracy, you cannot audit. Without auditability, regulated workloads cannot deploy. Make this metric part of the eval from day one, not a feature added at compliance-review time.

Reasoning accuracy. For DABstep-style downstream tasks where the agentic system’s output feeds into a reasoning step. This is a separate eval from extraction; conflating them produces misleading results. If your workflow has a reasoning layer, evaluate both extraction (with field-level F1) and downstream task success (with task-specific metrics) separately.

A general principle: pick one to three metrics that match your downstream cost, track them per-field where applicable, and resist the urge to add more. A team that evaluates on twelve metrics is implicitly saying it has no model of which errors matter most. Pick the metrics. Defend the choice. Track them religiously.

11.4 Regression testing

Vendor versions change. Even when the vendor name on your contract does not change, the model behind the API gets retrained, retuned, or rebuilt — and the new version sometimes behaves worse than the old version on your specific documents. Regression testing is the discipline that catches this before it ships.

The cadence that works in production:

Weekly: 50-document subset. A fast sanity-check on the highest-traffic document types. Run automatically after any vendor version-bump notification.
Monthly: full golden set. The full 200–500 document evaluation. Track per-field F1 over time. Look for drift in specific fields, not just aggregate.
Quarterly: golden-set refresh. Add new documents, retire stale ones. Re-baseline the per-field expectations on the refreshed set.
Vendor major-version bumps: full re-eval before promotion. Vendors that release major versions (LlamaParse v2 → v3, Reducto major-version updates, etc.) should not be promoted to production until the full golden set has been re-evaluated. This is the case where the rollback risk is highest.

Pin vendor versions in production. Most commercial vendors support version pinning; use it. The cost is missing out on improvements between pinned-version and current; the benefit is that production behavior changes only when you decide to upgrade, not when the vendor decides to roll out a new model overnight.

Track regression as a separate operational metric. A healthy regression cadence catches roughly one to three regressions per year on a stable golden set. Zero regressions per year is a sign your golden set is not sensitive enough; more than five per year suggests vendor instability that may itself be a procurement signal.

11.5 HITL feedback loops

The HITL queue is part of the eval. This is the point that most teams underweight, and it costs them.

Three production metrics live in the HITL queue:

Escalation rate. The percentage of documents the system routes to a human reviewer. Healthy 2026 deployments sit in the 1–10% range in steady state. Below 1% suggests the system is silently accepting errors that should have escalated; above 10% suggests extraction is too unreliable for the workload (or eval rules are too strict).

Acceptance rate. Of escalated documents, the percentage where the reviewer accepts the system’s extraction as-is (versus correcting it). Healthy range: 30–70%. Below 30% means the system is escalating documents it should have caught itself but the model still produced acceptable output; above 70% means the system is escalating documents it actually got right and wasting reviewer time.

Correction patterns. When reviewers correct, what do they correct? Track the per-field correction rate. A field that reviewers correct 40% of the time is a field the system handles poorly, regardless of what the per-field F1 on your golden set says. Production reviewer corrections are the most honest signal about model performance on novel documents — by definition, those documents were not in your golden set.

Closed-loop is the goal. Reviewer corrections should feed back into:

The golden set (high-disagreement documents become labeled examples).
Prompt tuning (systematic errors trigger prompt revisions).
Vendor selection (a vendor whose corrections cluster in your high-value fields is a worse fit than one whose corrections cluster in low-value fields, even at equal aggregate accuracy).
Schema design (corrections that flag fields as ambiguous suggest the schema may need a “needs review” sentinel rather than an aggressive default).

The anti-pattern is HITL as a dead-letter queue — reviewers correct documents one at a time, and the system never learns. This is the pattern that produces flat or declining accuracy over time while leadership wonders why the agentic-OCR investment is not paying off.

11.6 Grounding and auditability for compliance

For regulated workloads, grounding is not a feature, it is a requirement.

Practically, this means every extracted field carries metadata identifying the source region — page number, bounding box, optionally specific text span. The downstream consumer can render the output alongside the document with the cited region highlighted. The compliance auditor can verify the extraction. The HITL reviewer can correct it efficiently.

The eval implication: grounding accuracy is a first-class metric, not a checkbox. Measure it on the golden set. Track it over time. A vendor whose field-level F1 is 95% but whose grounding-accuracy is 70% is a worse fit for a compliance workload than a vendor at 92%/95% — the second vendor’s outputs are auditable; the first vendor’s are not.

The retention discipline matters too. Audit logs for grounded extractions need to persist for the regulatory retention window — six years for HIPAA, varying by jurisdiction for GDPR and SOX-equivalent regimes. Verify that the vendor’s audit-log retention matches your regulatory window before signing, not after the first audit.

For ZDR (Zero Data Retention) commitments — LandingAI ADE Team plan, Mistral self-hosted, certain Reducto configurations — verify in eval that ZDR is actually honored in production. The verification is operationally non-trivial: you cannot easily prove a negative (“the vendor did not retain my document”), but you can verify that the vendor’s audit logs and infrastructure are consistent with the ZDR claim. Some compliance regimes require a SOC 2 Type II report that specifically attests to ZDR; insist on this rather than the bare contractual claim.

11.7 Cost-aware evaluation

The metric most teams report — accuracy — is the wrong target. The metric that predicts the monthly bill is cost-per-correct-extraction-with-provenance, which combines accuracy with cost and HITL load:

\[ \text{cost-per-correct} = \frac{\text{API cost per page}}{\text{accuracy}} + \text{HITL cost per escalation} \times \text{escalation rate} \]

Worked example. A vendor at $0.012 per page with 90% accuracy and a 5% escalation rate at $3 per escalation:

API contribution: $0.012 ÷ 0.90 = $0.0133 per correct extraction.
HITL contribution: $3 × 0.05 = $0.15 per page; $0.15 / 0.90 = $0.167 per correct extraction.
Total: $0.180 per correct extraction.

A competing vendor at $0.008 per page with 82% accuracy and an 8% escalation rate at the same $3 per escalation:

API contribution: $0.008 ÷ 0.82 = $0.00976 per correct extraction.
HITL contribution: $3 × 0.08 = $0.24 per page; $0.24 / 0.82 = $0.293 per correct extraction.
Total: $0.303 per correct extraction.

The second vendor is 33% cheaper per page and 68% more expensive per correct extraction. The cost-per-page number is misleading; the cost-per-correct number is the actual answer.

Run this calculation on every vendor in your shortlist. Most teams find the ranking changes — sometimes substantially — relative to ranking by cost per page or accuracy alone. The combined metric is the one that should govern the decision.

A practical note on the HITL cost estimate. A reviewer who is a domain expert (a paralegal for legal documents, a nurse for clinical records) loads at roughly $50–$150 per hour. At three to five minutes per review, that translates to $2.50–$12.50 per escalation. Plug your team’s actual numbers in; the order of magnitude is robust, but the specific cost-per-correct number is sensitive to this input.

11.8 Production cost observability

Most teams treat cost as a finance problem rather than an evaluation problem. In agentic OCR specifically this is the wrong framing. Cost is volatile in this category in ways that traditional infrastructure cost is not: a vendor changes its pricing tiers, a model regression bumps the average retry budget, a misclassified batch flows through the most expensive premium tier instead of the standard one, a runaway query against the platform-tier products hits five figures in a single run. The failure modes are operational, not financial, and they belong in the observability stack.

Three things every production agentic OCR deployment should be tracking from week one.

11.8.1 Per-document cost attribution

Every processed document should carry an attached cost record. Not “we spent $X this month on parsing” — that’s a finance report. What’s needed is per-document cost broken down by step: classification cost, parsing cost (and which tier was invoked), retry cost (how many retries triggered, and at what unit price each), validation cost, HITL escalation cost where applicable.

The cost record is what lets you answer the question that matters in incidents: why did this document cost forty cents when the median is three? Without per-document attribution, the answer is unfindable. With it, the answer is usually “the reflection loop retried six times, each at the agentic-plus tier” — which is then a fixable failure mode.

The implementation: emit a structured cost event to your observability backend alongside every extraction. Most teams under-engineer this and discover they cannot trace cost back to documents when a budget alarm fires. Engineer it once, properly, before you need it.

11.8.2 Budget alerts and circuit breakers

Two thresholds worth defining explicitly:

Daily/monthly budget envelope. A soft alert at 60% of expected spend, a hard alert at 90%, an action threshold at 110%. The action threshold should trigger a circuit breaker — automatic throttling, rejection of low-priority work, or in the worst case a temporary pipeline pause — rather than continuing to burn through budget until someone notices on the bill.
Per-document cost ceiling. A single document should not cost more than 3–5× the median for its document class. When it does, something has gone wrong (retry-thrash, mismatched tier, classification error routing to a more expensive pipeline). The right response is to stop processing that document and escalate to HITL or a dead-letter queue rather than let the reflection loop burn an unlimited retry budget.

Both of these need to be implemented at the application layer, not relied on at the vendor layer. Vendors will happily process the runaway document and bill you for it. Your team’s spend discipline is your problem.

A 2026 incident worth knowing about: a Snowflake Cortex customer reported a single AI_PARSE_DOCUMENT query that ran up roughly $5,000 in credits when run against a large corpus without throttling (Seemore Data 2026). The pattern is common across all platform-tier products — the convenience of “just call this SQL function” hides how aggressively the function will scale when given an aggressive query. Circuit breakers are the cure.

11.8.3 Vendor-cost drift detection

Vendors change prices. Models change behavior. A model retrain at the vendor can shift the median number of input tokens per document — at frontier-VLM API pricing that translates directly to a higher cost per page even though the per-token rate did not change. The vendor’s pricing page still says the old number; your monthly bill says the new number.

Track three metrics weekly on a representative subset of production traffic:

Median cost per page, by document class and vendor tier.
Median cost per correct extraction (i.e., cost / accuracy on the golden set).
P95 cost per document — the long tail is where retry-thrash and runaway extractions hide.

A 10%+ shift week-over-week on any of these is a signal to investigate. Sometimes it is real (vendor pricing change, model retrain) and you adjust budgets. Sometimes it is a regression on your side (classification accuracy dropped, more documents now route to the expensive tier) and you fix it.

11.8.4 The cost / accuracy frontier

The denominator that should govern operational decisions — cost-per-correct-extraction-with-provenance — is itself a moving target. As vendors evolve and as your document distribution shifts, the optimal point on the cost / accuracy frontier moves. Teams that rebalance vendor mix or tier choice on this signal every quarter outperform teams that pick a configuration once and ride it until budgets force a panic.

Practical pattern: maintain a small “challenger” eval — 5–10% of production traffic routed to a different vendor or tier — and compare cost-per-correct against the current champion. Promote when the challenger wins on the joint metric; iterate. This is the same idea as champion/challenger experimentation in ML deployment, applied to vendor selection.

11.8.5 Runaway-cost failure modes

A short catalog of cost incidents the field has accumulated by mid-2026:

Retry-thrash on a difficult document class. The reflection loop burns its full retry budget per document because the model cannot converge on a particular layout. Per-document cost goes up 5–10×, escalation rate creeps up, monthly bill spikes. Fix: detect thrash early (per-document retry count > N triggers escalation to HITL rather than further retries).
Tier misrouting. A classification mistake sends documents to the premium tier when the standard tier was appropriate. Fix: cross-check classification confidence against tier cost — if classification is below 70%, default to the cheaper tier with HITL on the output rather than burning premium-tier cost on uncertain documents.
Vendor pricing changes mid-month. A vendor doubles a tier’s price; your budget assumptions are now wrong; the monthly bill is a surprise. Fix: weekly cost-per-page tracking, alerting on shifts.
Platform-tier “convenience runaway.” Single AI_PARSE_DOCUMENT-style calls run against unscoped corpora, billing thousands in credits. Fix: enforce row-limit or document-count caps at the orchestration layer, not at the SQL function level.
Frontier-VLM context-window inflation. A prompt change or document trend pushes input tokens up; cost rises silently because token-based pricing is invisible per-document until you aggregate. Fix: track input-token median per document class as a separate metric.

Each of these is preventable. None of them is automatically prevented by any vendor. Operational cost observability is the discipline that catches them before they catch you.

11.9 When your eval is the bottleneck

A few honest observations about eval that most vendor pitches will not surface:

If your eval is fast, your eval is suspicious. Real evals on 200–500 documents take minutes to tens of minutes to run end-to-end, depending on the vendor’s API and your eval rubric. An eval that completes in seconds is probably scoring against a too-small subset, a too-permissive rubric, or both.

If your eval is small (under 100 documents), it can’t reliably distinguish vendors within 5 percentage points. The statistical noise on a 50-document eval is large enough that a vendor scoring 87% and one scoring 82% are not reliably different. 200+ documents is the minimum for reliable cross-vendor comparison.

If your eval has no OOD slice, you don’t know how the system fails on novel documents. Hold out a slice of explicitly out-of-distribution documents — document types your production traffic has not yet seen, or vendor formats new to your pipeline — and measure separately. Most production incidents start as OOD failures; an eval that never tests OOD performance is silent about the most consequential failure mode.

Investment in eval is almost always under-funded relative to investment in model selection. The standard ratio is something like 10:1 in favor of evaluating vendors and against building the eval that makes vendor evaluation meaningful. Reverse this. Spend more time on the golden set, more time on metric selection, more time on regression cadence. The vendor decision becomes obvious once the eval is good.

11.10 What good eval looks like in production

A healthy 2026 agentic OCR deployment exhibits these eval properties simultaneously:

Golden set of 200–500 documents from production traffic, refreshed quarterly.
One to three metrics matched to the workload (typically: field-level F1, schema-validity, grounding accuracy).
Per-field tracking, not just aggregate.
Regression cadence — weekly subset, monthly full, quarterly refresh, vendor-version full re-eval.
HITL closed-loop: corrections flow back into the golden set, prompt tuning, vendor selection.
Cost-per-correct-extraction computed and reported as the primary economic metric.
OOD slice held out and measured separately to surface novel-document failures.

No deployment achieves all seven on day one. Teams that work toward them iteratively, over twelve to eighteen months, produce robust, defensible eval infrastructure. Teams that treat eval as a setup task to be completed and forgotten produce evaluations that gradually lose connection to production reality and start lying without warning.

Named takes

Golden sets are unfashionable, slow, and the single best ROI any agentic OCR team can deliver. Build one before you buy anything.

If your eval is faster than your inference, your eval is wrong. Real evals are not fast.

The HITL queue is part of the eval, not a fallback bolted on the side. The metric that tells you the most about your system in production lives in the reviewer’s accept/correct decisions.