Document - AI Ground Truth Beyond OCR: Reading Order, Tables, Fields and Page Relationships

Document AI systems do more than OCR: they need to recover the structure and relationships that make a page usable. For teams building AI training datasets, a model can transcribe every word correctly and still fail the workflow it was built for.

An invoice can contain the right amount but attach it to the wrong line item. A two - column report can contain every sentence while stitching the columns together in the wrong order. A table can preserve every cell value but place one value under the wrong header. These are document - understanding errors that character error rate or word error rate alone cannot capture.

That is why document AI ground truth needs to go beyond verified text. If the application depends on reading order, table topology, field relationships, or continuity across pages, the reference dataset needs labels for those relationships so each one can be evaluated separately.

The distinction is not that modern OCR is limited to plain text. Current systems such as Google Cloud Document AI can return OCR text, coordinates and layout signals. The more useful distinction is between text - recognition accuracy and structural accuracy: a transcription metric can tell you whether the words were recognized, but not whether the document relationships needed by the task were reconstructed correctly.

​

Document AI ground truth is verified reference data used to train or evaluate systems that convert documents into structured information. Depending on the task, it can include OCR text and layout regions, reading order, table structure, named fields, key - value relationships, document classes, page boundaries, and cross - page links.

Ground truth should match the output being judged. If a model is expected to recover a table hierarchy, pair a field with its value or join a table across a page break, verified transcription is necessary but not sufficient. The evaluation set needs to record the relationship itself.

This is also why the phrase “OCR accuracy” needs care. Character error rate (CER) and word error rate (WER) measure recognized text against a reference transcription. They are useful for OCR quality, but they do not tell you whether a correctly recognized amount was assigned to the right field or whether correctly recognized blocks were placed in the right sequence.

​

Reading order looks trivial on a single - column letter. It becomes a separate annotation problem as soon as a page contains two columns, a sidebar, a pull quote, a footnote, a floating caption or a form with several aligned groups.

The reference data therefore needs an explicit unit of ordering. A project might order blocks, regions, lines or another layout unit, but the rule needs to be fixed before production annotation begins. Otherwise two careful reviewers can disagree while both believe they are following “natural reading order.”

Recent research treats reading order as a measurable task rather than an informal by - product of layout detection. LaTeX2Layout, published at AAAI 2026, recovers both layout regions and reading order from compiler traces and evaluates sequence agreement with Kendall’s tau separately from element grounding. ROOR likewise annotates reading order as explicit ordering relations rather than assuming that region boxes imply sequence.

The same direction appears in newer visually rich document benchmarks. MosaicDoc, also published at AAAI 2026, combines OCR, reading - order, localization and document - question - answering annotations over complex newspaper and magazine layouts. The useful lesson is not a particular benchmark score:  locating the right regions and ordering those regions are different capabilities, so an evaluation set should not treat them as interchangeable.

If sequence matters to the downstream task, sequence needs its own reference labels.

​

Table extraction has at least two distinct failure modes. A model can misread the text inside a cell, or it can read the text correctly while assigning the cell to the wrong row, column or header visual hierarchy. A single blended accuracy number makes those failures hard to diagnose.

Tree Edit Distance - based Similarity (TEDS), introduced with PubTabNet, compares a predicted table representation with the reference as a tree. Structure - only variants commonly reported as TEDS - Struct or TEDS - S ignore cell text so that transcription errors do not dominate the structural score. Current 2026 table - extraction benchmarks still report TEDS and TEDS - S separately for exactly this reason.

This separation is operationally useful. If performance drops after a model change, a team needs to know whether the regression came from OCR, a missed spanning cell, a shifted column boundary or an incorrect header - body relationship. Those problems need different fixes.

Dataset domain matters too. PubTabNet was built from scientific - article tables, while FinTabNet was created from financial documents. A model that handles one population well may struggle when the layout grammar changes, particularly around nested headers, subtotals, spanning cells and dense financial formatting.

Suppose a form contains three occurrences of the label “Date”: invoice date, due date and delivery date. A model can locate every label and every date value, transcribe all six spans correctly, and still fail if it pairs them incorrectly.

That is why field extraction often needs relationship labels rather than boxes alone. The original FUNSD dataset makes this distinction explicit: it labels semantic entities such as questions and answers and separately records links between those entities. The annotation says not only “this text exists here” but also “these two pieces belong together.”

The difficult part is defining the unit that can be linked. A multi - line address could be one value, several lines or a set of nested fields. A vague schema lets each annotator make a different choice, which turns disagreement into ground - truth noise.

That risk is not theoretical. Researchers revisiting FUNSD documented inconsistent labels and revised the dataset for key - value detection. Later relabeling work has continued to highlight granularity and linking inconsistencies. The practical lesson is simple: define entity boundaries and linking rules with difficult examples before a large annotation team begins.

Model - assisted pre - labeling can reduce repetitive work. A model can propose boxes, fields, tables or links and a reviewer can correct them. But the proposal should remain a proposal: reviewers need to be trained to look for plausible - looking structural mistakes rather than assuming a confident model output is already ground truth.

A missing link is not the same defect as a missing field. The label and value can both be present and perfectly transcribed while the relationship between them is wrong.

​

A page is not always the right unit of evaluation. A table may start on page 4 and continue on page 5. A scanned batch may contain three invoices back - to - back because they arrived as one PDF. A reference in the body may point to an appendix several pages later. A section heading may establish a hierarchy that continues across page breaks.

Those relationships need document - level labels if the application depends on them. Useful examples include:

• Table continuation  -  whether two fragments belong to the same logical table and how the continuation should be joined.

• Document boundaries  -  where one logical document ends and the next begins inside a combined scan or package.

• Cross - references  -  which figure, table, section or note a reference actually points to.

• Hierarchical structure  -  how headings, subheadings and body sections nest across the full document.

This is harder to annotate consistently than drawing boxes on isolated pages. Reviewers need enough context to understand the whole document, and the work is less easy to divide across people without losing continuity.

Public benchmarks increasingly reflect that reality. DU​DE was designed around multi - page, multi - domain visually rich documents, while VRDU emphasizes rich schemas, complex templates, repeated and nested fields, and layout diversity in business documents. The point for a production dataset is not to imitate one benchmark. It is to decide which cross - page relationships the actual application depends on and label those relationships deliberately.

​

Separate metrics are not academic housekeeping. They make failures diagnosable.

Imagine a field - extraction system gets worse after a model update. If the only result is “92% accurate,” the number does not tell the team where to look. The regression could come from text recognition, reading order, table structure, field linking, document boundaries or several of them at once.

A useful evaluation set keeps those signals distinct enough to investigate. That does not mean every project needs a dozen dashboard metrics. It means the reported metric should match the claim being made. If the target is transcription, report a transcription metric. If the target is table structure, report a structure metric. If the target is key - value extraction, define how entity detection and linking are both scored.

If two errors require different fixes, they should normally remain distinguishable in evaluation. A summary score can still be useful, but it should not replace the component measures needed to diagnose what changed.

A quality metric without its measurement method is not evidence. “95% accurate” only becomes useful when a reader knows what counted as correct, which reference labels were used and how ambiguous cases were resolved.

​

The same words can create a very different document - understanding problem when the layout changes. That makes controlled design variation a useful way to test whether a model has learned document structure or only familiar templates.

One practical experiment is to hold the content constant and change one layout variable at a time:

• single - column text versus two columns;
• a field value placed next to its label versus aligned at a distance;
• a simple table header versus merged or nested headers;
• a caption directly under an image versus a floating callout;
• a table kept on one page versus the same table continued onto a second page.

Run the variants through the same OCR/document AI pipeline and score transcription and structure separately. If the words remain correct while reading order, table topology or field links deteriorate, the test has isolated a layout - generalization problem rather than an OCR problem.

For a design - focused publication, this is a more credible information - gain angle than another generic list of document AI benefits. It turns page design into a controlled variable that can be measured. Do not publish numerical findings unless the experiment is actually run and its methodology is documented.

A project does not need every layer by default. It needs the layers that correspond to real downstream decisions. A document - classification system may not need cell - level table labels. An invoice automation system probably does. The goal is not maximum annotation density; it is sufficient reference data to judge the tasks that actually matter.

​

The expensive mistakes usually happen before the first production batch is labeled. A workable pipeline looks like this:
1. Define the schema first.Decide what counts as a field, table, reading - order unit and document boundary. Include difficult examples rather than only ideal pages.

2. Run a pilot batch. Use a small but varied sample to find disagreements in the guidelines before they spread across thousands of pages.

3. Measure disagreement where ambiguity matters. Not every label needs a formal inter - annotator agreement statistic, but recurring disagreement should trigger a guideline change or adjudication rule rather than silent averaging.
4. Use model - assisted pre - labeling selectively. Automation is useful for repetitive proposals, while human attention should concentrate on relationships and edge cases that are costly to get wrong.
5. Score content and structure separately. Keep transcription, table topology, entity linking and cross - page relationships distinct enough to diagnose regressions.
6. Version the schema and preserve provenance. A team should be able to tell which guideline version produced a label, how it was reviewed and why an exception was accepted.

The smallest useful schema is better than an elaborate schema nobody can apply consistently. Define only the labels that answer real model or product questions, then make those labels reproducible.

​

If you are licensing, commissioning or approving a document dataset, ask questions that expose the structure behind the headline accuracy number:
• What is the unit of reading - order annotation: words, lines, blocks or regions?
• Are table structure and cell text evaluated separately?
• Are key - value relationships explicitly labeled, or inferred from proximity?
• How are repeated, nested and multi - line fields handled?
• Are continuation and document - boundary labels available across pages?
• Which ambiguous cases are documented in the annotation guidelines?
• Was there a pilot and an adjudication process for reviewer disagreement?
• Can the dataset version be traced to a schema version and production batch?
• Which metrics are reported, and what exactly counts as a correct prediction?

Those questions often reveal more about dataset fitness than one aggregate accuracy figure.

If you are still at the earlier sourcing stage, DesignWizard's guide to finding professional datasets for machine learning projects covers broader dataset discovery and sourcing. This article starts one step later, once the question becomes what the reference labels themselves need to contain.

For teams commissioning document data, Wavebreak Media's document datasets for AI training page covers licensed and custom document datasets for OCR, layout understanding, field extraction, tables, and structured metadata. For commercial projects, its AI dataset licensing guidance also covers usage rights, provenance, releases, and supporting documentation. A document AI project should still define its own structural schema and acceptance criteria rather than assuming that the word “annotated” covers the relationships it needs.

What is documented AI ground truth?

Document AI ground truth is verified reference data used to train or evaluate systems that interpret documents. Depending on the task, it can include transcription, reading order, table structure, field relationships, document boundaries and cross - page links.

What is the difference between OCR accuracy and document AI accuracy?

OCR metrics such as CER and WER measure whether text was recognized correctly. Document AI evaluation may also need to measure whether regions were ordered correctly, tables were reconstructed correctly, fields were linked to the right values and relationships across pages were preserved.

What annotations does a document AI dataset need?

Only the annotations required by the downstream task. A form extractor may need key - value links and repeated - field rules; a table system may need row, column, span and header annotations; a multi - page workflow may need document boundaries and continuation labels.

How do you evaluate reading order in a document?

First define the unit being ordered, such as blocks or regions, then compare the predicted sequence with a reference sequence. Research benchmarks use ranking measures such as Kendall’s tau, but the appropriate metric depends on the annotation and task.

How is table structure accuracy measured separately from OCR?

Metrics such as TEDS - Struct or TEDS - S compare table structure while excluding cell text. That allows structural errors such as wrong row/column membership or missed spans to be evaluated separately from transcription mistakes.

Can an existing OCR dataset be reused for document AI evaluation?

Sometimes, but usually only for the layers it already labels. If the application depends on reading order, table structure, field links or cross - page relationships, those labels may need to be added before the dataset can evaluate the full task.

How do you evaluate multi - page document AI systems?

Use document - level references for the relationships that cross page boundaries: table continuation, document boundaries, hierarchical sections and cross - references. Per - page accuracy alone cannot show whether those relationships were preserved.

What should a buyer check before commissioning document annotation?

Ask which relationships are explicitly labeled, how difficult cases are defined, how reviewers resolve disagreement, which QA process is used and what every reported metric actually measures.

​

The easiest document AI error to measure is a wrong character. Many of the expensive failures happen after the characters are already correct.

A table cell sits under the wrong header. A value is linked to the wrong label. Two columns are read in the wrong order. A table continuation is treated as a new table because the evaluation set never recorded that the pages belonged together.

Good ground truth makes those relationships visible. Text accuracy tells you what was recognized; structural ground truth tells you whether the output was organized in the way the task requires.

That is the useful dividing line for evaluation: if a downstream decision depends on a relationship, the dataset specification should say how that relationship is represented, reviewed and scored.

​