Document AI systems need more than accurate text recognition. They must also understand how information is arranged and connected within a document.

For example, an invoice processing model may read every word correctly but assign a payment amount to the wrong line item. A model may also extract all the text from a report but read its columns in the wrong order.

These errors show why OCR accuracy alone is not enough to evaluate document AI systems. The ground truth dataset must also capture the structure and relationships that the model is expected to identify.

This guide explains how to create and evaluate document AI ground truth for reading order, table structure, key-value extraction, and multi-page document relationships. It also provides practical checks for teams developing, evaluating, or sourcing document AI training datasets.

Modern OCR systems can already return text, coordinates, and layout information. The key difference is whether the dataset provides verified labels for the specific relationships a model needs to reconstruct and whether those relationships are evaluated correctly.

Infographic comparing correct and incorrect invoice data extraction, showing accurate mapping of invoice number, date, company details, billing address, and line-item amounts on the left, versus misaligned fields and mismatched service costs on the right.

What Is Document AI Ground Truth?

Document AI ground truth is verified reference data used to train or evaluate AI systems that process documents. It can include correct text, page layout, reading order, table structure, key-value relationships, document boundaries, and links between pages. The exact annotations depend on what the AI system is expected to do.

​Ground truth should match the output being judged. If a model is expected to recover a table hierarchy, pair a field with its value or join a table across a page break, verified transcription is necessary but not sufficient. The evaluation set needs to record the relationship itself.

This is also why the phrase “OCR accuracy” needs care. Character error rate (CER) and word error rate (WER) measure recognized text against a reference transcription. They are useful for OCR quality, but they do not tell you whether a correctly recognized amount was assigned to the right field or whether correctly recognized blocks were placed in the right sequence.

​

Evaluation layer
Reference data
Failure it exposes

Text recognition

Characters, words and text regions

Wrong or missing text

Reading order

The intended sequence of blocks, lines or regions

Correct text assembled in the wrong sequence

Table structure

Rows, columns, spans and header relationships

Correct cell text placed in the wrong structure

Field relationships

Which labels, values and entities belong together

Correct spans paired with the wrong counterpart

Document relationships

Continuations, boundaries and cross-page references

Pages or tables split, merged or truncated incorrectly

Example: How Incorrect Ground Truth Can Affect Invoice AI

Imagine an invoice contains the following information:

Invoice number: INV-1001

Item A: $100

Item B: $200

Total: $300

An AI model may recognize all the text correctly but assign the $200 amount to Item A instead of Item B.

The OCR output may be correct, but the structured result is wrong.

A useful ground truth dataset should therefore record both the correct text and the correct relationships between item names, amounts, and totals.

This example shows why document AI evaluation should measure text recognition and structural accuracy separately.

Four-layer document AI diagram showing raw pixels, OCR text and layout, structural elements such as reading order and table relationships, and document-level connections including page relationships, continuations, and cross-references.

Reading Order: Correct Text Can Still Be in the Wrong Sequence

Reading order looks trivial on a single - column letter. It becomes a separate annotation problem as soon as a page contains two columns, a sidebar, a pull quote, a footnote, a floating caption or a form with several aligned groups.

The reference data therefore needs an explicit unit of ordering. A project might order blocks, regions, lines or another layout unit, but the rule needs to be fixed before production annotation begins. Otherwise two careful reviewers can disagree while both believe they are following “natural reading order.”

Recent research treats reading order as a measurable task rather than an informal by - product of layout detection. LaTeX2Layout, published at AAAI 2026, recovers both layout regions and reading order from compiler traces and evaluates sequence agreement with Kendall’s tau separately from element grounding. ROOR likewise annotates reading order as explicit ordering relations rather than assuming that region boxes imply sequence.

The same direction appears in newer visually rich document benchmarks. MosaicDoc, also published at AAAI 2026, combines OCR, reading - order, localization and document - question - answering annotations over complex newspaper and magazine layouts. The useful lesson is not a particular benchmark score:  locating the right regions and ordering those regions are different capabilities, so an evaluation set should not treat them as interchangeable.

If sequence matters to the downstream task, sequence needs its own reference labels.


​

Table Ground Truth: Separate Structure From Cell Text

Table extraction has at least two distinct failure modes. A model can misread the text inside a cell, or it can read the text correctly while assigning the cell to the wrong row, column or header visual hierarchy. A single blended accuracy number makes those failures hard to diagnose.

Tree Edit Distance - based Similarity (TEDS), introduced with PubTabNet, compares a predicted table representation with the reference as a tree. Structure - only variants commonly reported as TEDS - Struct or TEDS - S ignore cell text so that transcription errors do not dominate the structural score. Current 2026 table - extraction benchmarks still report TEDS and TEDS - S separately for exactly this reason.

This separation is operationally useful. If performance drops after a model change, a team needs to know whether the regression came from OCR, a missed spanning cell, a shifted column boundary or an incorrect header - body relationship. Those problems need different fixes.

Dataset domain matters too. PubTabNet was built from scientific - article tables, while FinTabNet was created from financial documents. A model that handles one population well may struggle when the layout grammar changes, particularly around nested headers, subtotals, spanning cells and dense financial formatting.

Table ground truth requirement
What it captures
What it does not capture

Cell text (OCR layer)

Transcription inside each cell

Row/column membership, merges and header hierarchy

Row and column structure

Which cells belong to each row or column, including spans

Whether the cell text itself is correct

Header hierarchy

Multi-row, nested and spanning headers

The semantic meaning of every value

TEDS-Struct / TEDS-S

Similarity of table structure with text excluded

OCR accuracy, which should be measured separately

Infographic comparing correct and incorrect table structures in document AI, showing a sales table with merged headers for Q1 and Q2, regional data, units, revenue, and subtotals. Red highlights identify structural errors, including a split merged header and a subtotal shifted into the wrong column.

Field and Key - Value Ground Truth: Finding a Value Is Not the Same as Linking It

Suppose a form contains three occurrences of the label “Date”: invoice date, due date and delivery date. A model can locate every label and every date value, transcribe all six spans correctly, and still fail if it pairs them incorrectly.

That is why field extraction often needs relationship labels rather than boxes alone. The original FUNSD dataset makes this distinction explicit: it labels semantic entities such as questions and answers and separately records links between those entities. The annotation says not only “this text exists here” but also “these two pieces belong together.”

The difficult part is defining the unit that can be linked. A multi - line address could be one value, several lines or a set of nested fields. A vague schema lets each annotator make a different choice, which turns disagreement into ground - truth noise.

That risk is not theoretical. Researchers revisiting FUNSD documented inconsistent labels and revised the dataset for key - value detection. Later relabeling work has continued to highlight granularity and linking inconsistencies. The practical lesson is simple: define entity boundaries and linking rules with difficult examples before a large annotation team begins.

Model - assisted pre - labeling can reduce repetitive work. A model can propose boxes, fields, tables or links and a reviewer can correct them. But the proposal should remain a proposal: reviewers need to be trained to look for plausible - looking structural mistakes rather than assuming a confident model output is already ground truth.

A missing link is not the same defect as a missing field. The label and value can both be present and perfectly transcribed while the relationship between them is wrong.

​

Page and Document Relationships: The Errors Single - Page QA Cannot See

Document AI specialist annotating a newspaper article with numbered reading-order markers, colored bounding boxes, and structural notes while reviewing the document layout in an annotation interface on a laptop.

A page is not always the right unit of evaluation. A table may start on page 4 and continue on page 5. A scanned batch may contain three invoices back - to - back because they arrived as one PDF. A reference in the body may point to an appendix several pages later. A section heading may establish a hierarchy that continues across page breaks.

Those relationships need document - level labels if the application depends on them. Useful examples include:

• Table continuation  -  whether two fragments belong to the same logical table and how the continuation should be joined.

• Document boundaries  -  where one logical document ends and the next begins inside a combined scan or package.

• Cross - references  -  which figure, table, section or note a reference actually points to.

• Hierarchical structure  -  how headings, subheadings and body sections nest across the full document.

This is harder to annotate consistently than drawing boxes on isolated pages. Reviewers need enough context to understand the whole document, and the work is less easy to divide across people without losing continuity.

Public benchmarks increasingly reflect that reality. DU​DE was designed around multi - page, multi - domain visually rich documents, while VRDU emphasizes rich schemas, complex templates, repeated and nested fields, and layout diversity in business documents. The point for a production dataset is not to imitate one benchmark. It is to decide which cross - page relationships the actual application depends on and label those relationships deliberately.


​

Diagram illustrating document-level relationships across five pages, showing a table continuing from page 3 to page 4 and a cross-reference marked [1] on page 2 resolving to a source entry on page 5.

Why These Layers Should Be Scored Separately

Separate metrics are not academic housekeeping. They make failures diagnosable.

Imagine a field - extraction system gets worse after a model update. If the only result is “92% accurate,” the number does not tell the team where to look. The regression could come from text recognition, reading order, table structure, field linking, document boundaries or several of them at once.

A useful evaluation set keeps those signals distinct enough to investigate. That does not mean every project needs a dozen dashboard metrics. It means the reported metric should match the claim being made. If the target is transcription, report a transcription metric. If the target is table structure, report a structure metric. If the target is key - value extraction, define how entity detection and linking are both scored.

If two errors require different fixes, they should normally remain distinguishable in evaluation. A summary score can still be useful, but it should not replace the component measures needed to diagnose what changed.

A quality metric without its measurement method is not evidence. “95% accurate” only becomes useful when a reader knows what counted as correct, which reference labels were used and how ambiguous cases were resolved.

​

Why Layout Design Changes the Evaluation Problem

The same words can create a very different document - understanding problem when the layout changes. That makes controlled design variation a useful way to test whether a model has learned document structure or only familiar templates.

One practical experiment is to hold the content constant and change one layout variable at a time:

• single - column text versus two columns;
• a field value placed next to its label versus aligned at a distance;
• a simple table header versus merged or nested headers;
• a caption directly under an image versus a floating callout;

• a table kept on one page versus the same table continued onto a second page.

Run the variants through the same OCR/document AI pipeline and score transcription and structure separately. If the words remain correct while reading order, table topology or field links deteriorate, the test has isolated a layout - generalization problem rather than an OCR problem.

For a design - focused publication, this is a more credible information - gain angle than another generic list of document AI benefits. It turns page design into a controlled variable that can be measured. Do not publish numerical findings unless the experiment is actually run and its methodology is documented.

​


A Practical Ground Truth Checklist by Layer

Layer
Ground truth artifact
Common failure if skipped

Text

Verified transcription plus coordinates where needed

Wrong characters or missed text regions

Reading order

Ordered block, line or region labels

Correct text assembled in the wrong sequence

Tables

Row, column, span and header structure separate from cell text

Values assigned to the wrong row or column

Fields

Explicit key-value or entity-link annotations

Correct spans paired with the wrong counterpart

Pages

Continuation, boundary and cross-reference labels

Tables or documents silently split, merged or truncated

A project does not need every layer by default. It needs the layers that correspond to real downstream decisions. A document - classification system may not need cell - level table labels. An invoice automation system probably does. The goal is not maximum annotation density; it is sufficient reference data to judge the tasks that actually matter.

​

How to Build Document AI Ground Truth Without Annotation Debt

Many annotation problems begin before the first production batch is labeled. A clear workflow helps teams reduce inconsistencies and avoid expensive rework.


1. Define the schema first. Decide what counts as a field, table, reading-order unit, and document boundary. Include difficult examples in the guidelines so reviewers know how to handle them.

2. Run a pilot batch. Use a small but varied sample to find disagreements in the guidelines before they spread across thousands of pages.

3. Measure disagreement where ambiguity matters. Not every label needs a formal inter - annotator agreement statistic, but recurring disagreement should trigger a guideline change or adjudication rule rather than silent averaging.
4. Use model - assisted pre - labeling selectively. Automation is useful for repetitive proposals, while human attention should concentrate on relationships and edge cases that are costly to get wrong.
5. Score content and structure separately. Keep transcription, table topology, entity linking and cross - page relationships distinct enough to diagnose regressions.
6. Version the schema and preserve provenance. A team should be able to tell which guideline version produced a label, how it was reviewed and why an exception was accepted.

The smallest useful schema is better than an elaborate schema nobody can apply consistently. Define only the labels that answer real model or product questions, then make those labels reproducible.

​

How to Evaluate a Document AI Dataset Before You Buy or Commission It

If you are licensing, commissioning or approving a document dataset, ask questions that expose the structure behind the headline accuracy number:

• What is the unit of reading - order annotation: words, lines, blocks or regions?
• Are table structure and cell text evaluated separately?
• Are key - value relationships explicitly labeled, or inferred from proximity?
• How are repeated, nested and multi - line fields handled?
• Are continuation and document - boundary labels available across pages?
• Which ambiguous cases are documented in the annotation guidelines?
• Was there a pilot and an adjudication process for reviewer disagreement?
• Can the dataset version be traced to a schema version and production batch?
• Which metrics are reported, and what exactly counts as a correct prediction?

Those questions often reveal more about dataset fitness than one aggregate accuracy figure. If you are still at the earlier sourcing stage, DesignWizard's guide to finding professional datasets for machine learning projects covers broader dataset discovery and sourcing. This article starts one step later, once the question becomes what the reference labels themselves need to contain.

When sourcing document AI training data, teams should check whether the dataset includes the annotations their models need. Wave Break Media's document datasets for AI training resource is relevant for teams exploring document data for OCR, layout understanding, table extraction, and other document processing tasks. Before selecting a dataset, confirm that its annotation schema, quality checks, and usage rights match your project's requirements.

Additional Checks Before Selecting a Dataset

Beyond annotation quality, buyers should also review:

✔ Document coverage: Does the dataset include the document types, layouts, languages, and formats used in your application?

✔ Data quality: How are incorrect, incomplete, or inconsistent annotations identified and corrected?

✔ Data provenance: Is the source of the data documented?

✔ Licensing: Does the license permit your intended training, evaluation, and commercial use?

✔ Data leakage: Are training and evaluation samples separated to reduce the risk of testing on documents the model has already seen?

✔ Format and delivery: Can the dataset be delivered in a format that works with your model pipeline?

✔ Quality acceptance criteria: Are the expected annotation quality levels and review requirements clearly defined?

A dataset is suitable only when its content, annotation quality, licensing, and evaluation design match the intended use case.


​

​

Frequently Asked Questions About Document AI Ground Truth

Document AI specialist reviewing a multi-page financial report with annotated page order, table relationships, cross-references, and continuation markers, using a laptop-based document review interface to validate document structure and quality.

What is Document AI ground truth?

Document AI ground truth is verified reference data used to train or evaluate systems that interpret documents. Depending on the task, it can include transcription, reading order, table structure, field relationships, document boundaries and cross - page links.

What is the difference between OCR accuracy and document AI accuracy?

OCR accuracy measures how correctly a system recognizes text. Document AI evaluation may also measure whether the text is placed in the correct reading order, assigned to the right table cells, linked to the correct fields, and connected across pages. The evaluation metrics should match the task being tested.

What annotations does a document AI dataset need?

A document AI dataset needs the annotations required by its intended task. Form extraction may require key-value links, table extraction may require row and column structure, and multi-page document processing may require page boundaries and continuation labels.

How do you evaluate reading order in a document?

First define the unit being ordered, such as blocks or regions, then compare the predicted sequence with a reference sequence. Research benchmarks use ranking measures such as Kendall’s tau, but the appropriate metric depends on the annotation and task.

How is table structure accuracy measured separately from OCR?

Metrics such as TEDS - Struct or TEDS - S compare table structure while excluding cell text. That allows structural errors such as wrong row/column membership or missed spans to be evaluated separately from transcription mistakes.

Can an existing OCR dataset be reused for document AI evaluation?

Yes, if the dataset contains the reference labels needed for the task. An OCR dataset may support text recognition evaluation, but additional annotations may be required to evaluate reading order, table structure, field relationships, or multi-page document understanding.

How do you evaluate multi - page document AI systems?

Use document - level references for the relationships that cross page boundaries: table continuation, document boundaries, hierarchical sections and cross - references. Per - page accuracy alone cannot show whether those relationships were preserved.

What should a buyer check before commissioning document annotation?

Ask which relationships are explicitly labeled, how difficult cases are defined, how reviewers resolve disagreement, which QA process is used and what every reported metric actually measures.

​

Final Takeaway

Accurate text recognition is only one part of document AI evaluation. A model must also understand how information is arranged, connected, and carried across pages.

For example, a model may recognize every word in an invoice but assign an amount to the wrong item. It may also extract a table correctly at the text level while placing values under the wrong headers.

These errors can only be evaluated when the ground truth dataset includes the necessary structural annotations.

The most effective approach is to define the required annotations before data collection, establish clear review guidelines, and evaluate each important task separately.

For teams developing or sourcing document AI datasets, the goal is simple: create reference data that accurately reflects how the AI system is expected to process real documents.

Jen Togonon

Jen Togonon

Jen Togonon is a digital content professional with experience in website content, online publishing, and creative digital projects. She enjoys creating useful and engaging content for online audiences.