How to Prevent Train-Test Leakage in Image and Video Datasets

Train-test leakage in machine learning occurs when information from a test or validation dataset influences model training or evaluation. This can make a model appear more accurate than it would be when processing genuinely new data. In image and video datasets, leakage can happen when duplicate images, edited versions of the same source, or related video frames appear in both training and testing sets.

Data leakage in machine learning happens when information that should be unavailable during model development influences training, preprocessing, model selection, or evaluation. It can make validation or test performance look stronger than real - world performance because the evaluation boundary has already been compromised.

For teams building models with licensed image and video datasets for AI training, one of the easiest forms of leakage to miss is train - test contamination through related visual content. Different file names and asset IDs can still represent the same image at another resolution, a crop or edit of the same source, adjacent frames from one recording, or observations that share an identity or production session that was supposed to remain unseen.

The practical issue is independence. A test split is only useful when it measures the kind of generalization the system will need after deployment, rather than rewarding the model for seeing a closely related version of the same underlying source during training.

An unseen asset ID does not necessarily mean an unseen visual source.

Data leakage occurs when information that should remain outside model development affects the trained model or the decisions used to build it. The result is usually an evaluation score that does not represent the intended deployment setting. In this article, “data leakage” refers to a machine - learning evaluation problem, not the cybersecurity meaning of unauthorized data disclosure.

Train - test contamination occurs when training information crosses into validation or test data, or vice versa. Exact duplicate records are the simplest example, but the same problem can arise when nominally different samples share a source, identity, session, sequence, or other dependency that the evaluation was supposed to hold out.

Preprocessing leakage occurs when a transformation is learned using information from data that should have remained held out. Common examples include fitting normalization parameters, imputers, feature selection, or other learned preprocessing on the complete dataset before evaluation instead of fitting them on training data only.

Target leakage occurs when model inputs contain information about the outcome that would not legitimately be available when the prediction is made.

The main focus here is train - test contamination in image and video datasets: keeping evaluation data genuinely independent from training data while preserving the relationships needed to audit that boundary.

Data leakage is also different from ordinary overfitting. A model can overfit a perfectly clean training set. Leakage compromises the evaluation boundary itself, so the reported test metric can become an unreliable estimate of generalization. For a broader discussion of this problem in scientific machine learning, see Kapoor and Narayanan’s Leakage and the reproducibility crisis in machine learning based science.

​

Image and video datasets create unique train-test leakage risks because different files do not always represent independent observations.

For example, a single photograph may exist as a high-resolution original, a resized version, a cropped image, and an edited copy. Although these files have different names, they may contain nearly identical visual information.

Video datasets present an additional challenge. A single recording can produce multiple clips or thousands of extracted frames. If related frames are randomly distributed across training and testing sets, a model may encounter familiar visual content during evaluation.

These relationships are especially important when building computer vision datasets for AI training. The dataset must reflect the type of new data the model is expected to handle after deployment.

Choose the split unit from the generalization claim first, then randomize at that unit if randomization is appropriate.

Group - aware validation tools implement the same principle computationally. For example, scikit - learn’s GroupShuffleSplit lets samples share a group label so that a group is held out as a unit when creating train - test partitions.

It is useful to distinguish three problems because they require different evidence.

1. Exact duplicates

These are byte - identical files. A cryptographic hash is usually sufficient. If two files produce the same strong content hash, they can be treated as identical for ordinary dataset - deduplication purposes. This check is cheap and should generally be part of ingestion or dataset QA. It is also the easiest case.

2. Near-duplicate visual variants

Near-duplicate images are different files that contain almost the same visual content. They may have been resized, compressed, cropped, rotated, or edited.

Unlike exact duplicates, these files will not produce the same cryptographic hash. Perceptual hashing can help identify images that look similar despite changes to their file data.

However, perceptual hashing is not perfect. Large crops, overlays, and significant edits may cause it to miss related images. Loose similarity thresholds may also group images that are actually different.

For this reason, teams should test their detection thresholds against representative examples from the dataset before using them to define evaluation groups.

3. Related or dependent samples

The hardest category is not necessarily duplicate content at all. Two photographs may show the same person from different viewpoints. Two clips may come from the same recording but contain different moments. Two photographs of a product may be visually distinct while sharing the same controlled studio setup. Embedding retrieval can help surface these relationships because learned representations can retrieve visually or semantically related assets that simple hashes miss. But similarity is only evidence. A high cosine - similarity score does not by itself prove that two records belong in one leakage group. The correct decision depends on the embedding model, threshold, available metadata, source lineage, and, most importantly, the evaluation objective.

Duplicate and Derivative Files Cross the Split

A source photograph may exist as a full - resolution file, resized delivery, edited version, crop, or re - export.

Video creates even more derivative relationships: multiple encodes, vertical and horizontal crops, extracts, previews, trimmed versions, and frame sequences.

If derivatives receive independent asset IDs and the split is performed at the ID level, different representations of one source can appear in training and testing.

Related Images from the Same Shoot Cross the Split

A photoshoot can produce tens or hundreds of related frames.

Whether they must remain together depends on the task. If the evaluation is supposed to measure generalization to new sessions, then randomly distributing frames from one session creates an unrealistically easy test. The same applies to repeated takes from one controlled video shoot.

Adjacent Video Frames Cross the Split

Video frames are temporally correlated by construction. If frames are extracted densely and then shuffled independently, frames separated by a fraction of a second can land in different partitions. For frame - level datasets, preserve the source video, sequence position, and timestamp instead of treating every extracted image as an unrelated example.

Related Assets Appear in Multiple Collections

Large media libraries are often assembled over time. The same underlying content may arrive through separate acquisitions or exist in both an original archive and a later derivative collection. If source lineage is incomplete, a per - collection deduplication pass may not discover the cross - collection relationship.

Visual retrieval can therefore be useful across the full candidate corpus, not just within one supplier or folder.

Public Benchmarks Overlap with Training Data

A separate problem arises when evaluation assets are publicly available and a model has been pretrained on large external corpora.

A 2025 ICCV Workshops study, Data Leakage in Visual Datasets, examined overlap within and between visual datasets and used image retrieval to show that all analyzed datasets presented some form of leakage; the authors also found that leakage compromised downstream evaluation reliability.

This kind of contamination is harder to rule out when the evaluator has no access to a model's pretraining corpus. A clean fine - tuning split therefore does not prove that a public benchmark was unseen during pretraining.

​

Preventing train-test leakage requires more than removing duplicate files. Teams must first define what the model should generalize to, identify related data, and keep those relationships within the appropriate dataset split.

The following workflow provides a practical way to create and audit image and video dataset splits.

​

1. Define the evaluation claim

Write down what the model is expected to generalize to: new files, recording sessions, people, locations, cameras, time periods, or another factor. That decision determines which relationships must remain isolated.

2. Preserve source lineage

Use existing production information before trying to reconstruct every relationship with computer vision. Useful fields can include original source ID, derivative - of ID, shoot or session ID, recording ID, contributor or acquisition source, participant or subject ID where appropriate and lawfully available, timestamps and frame position, and edit or export relationships. Lineage is often stronger evidence of dependency than visual similarity alone.

3. Detect duplicates and related visual content

Use multiple methods to identify potentially related files.

• Cryptographic hashes:  Find files that are exactly identical.

• Perceptual hashes:  Identify images that remain visually similar after resizing, compression, or minor edits.

• Image embeddings:  Retrieve visually or semantically related images that may not be detected by simpler methods.

• Source metadata:  Identify files that belong to the same shoot, recording, or derivative chain.

These methods should be used together. Similarity scores can help identify candidate matches, but they do not automatically prove that two files belong in the same evaluation group. Review ambiguous matches using metadata and the intended evaluation objective.

4. Build groups conservatively

Do not automatically merge every asset connected by any similarity edge. If image A is similar to B and B is similar to C, that does not prove that A and C should share one evaluation group. Decide which relationships are legitimately transitive. Combine source metadata, visual evidence, and manual review for ambiguous cases. Inspect unusually large clusters because they can reveal an overly permissive threshold or semantic chaining.

5. Assign groups, not individual records

Once the relevant groups have been defined, assign whole groups to training, validation, or testing. The group might be an exact - duplicate family, source recording, shoot, participant, location, or another task - specific unit. Randomization can still be used; it simply happens at the appropriate unit.

6. Keep learned preprocessing inside the training boundary

Grouping and fixed duplicate detection before split assignment should not be confused with fitting a model transformation on the entire dataset. Once the evaluation partitions are established, preprocessing that learns parameters from data should be fitted using training data only and then applied to validation or test data as appropriate. The protected test set should not become an iterative model - development tool.

7. Freeze and version the evaluation set

A split should be reproducible. Keep a versioned manifest containing stable asset identifiers, group identifiers, exclusion records, reference labels, and the split assignment. If the library changes, do not silently change the benchmark. Decide whether a new acquisition belongs in training, a future evaluation refresh, or neither.

8. Audit the boundary after splitting

After the split has been created, query test assets against the training index. Inspect the strongest cross - split matches and record why each candidate is or is not considered problematic. The audit is evidence that the split was checked; a similarity score by itself is not.

Useful audit outputs include:

• exact cross - split matches

• near - duplicate candidates

• shared - source relationships

• shared identities where identity isolation is required

• sequence or frame adjacency

• unexpected cluster sizes

• manual - review results

• unresolved limitations

The practical workflow is usually a combination rather than a competition between methods. A byte hash is cheap enough to use broadly. Perceptual hashing provides another inexpensive candidate layer. Embedding retrieval can search relationships that those methods miss. Source metadata can then confirm relationships that pixels alone cannot establish.

​

Video datasets need additional leakage controls because a single recording can generate many related clips and extracted frames.

A useful way to organize video data is:

Production session → Source recording → Derived clip → Extracted frame

Each level represents a relationship between the original recording and its derived assets.

For example, if a model is expected to generalize to new recordings, all clips and extracted frames from the same source recording should remain in the same dataset split.

If the model is expected to recognize new people, identity may also need to be used as a grouping factor. The correct split depends on what the model is expected to encounter after deployment.

The consequences are not hypothetical. In the CVPR 2026 paper Data Leakage Detection and De - duplication in Large Scale Geospatial Image Datasets, researchers studying the AICrowd Mapping Challenge dataset reported that nearly 90% (about 250,000) of training images had identical copies and that approximately 56,000 of 60,000 validation images were also present in training, amounting to 93% train - validation leakage.

That is an extreme case, not a general estimate for image datasets. Its value is in showing how a benchmark can look structurally plausible while containing enough redundancy to make evaluation results misleading.

More broadly, the 2025 Data Leakage in Visual Datasets study found leakage in all of the visual datasets it analyzed and reported that leakage, from severe to subtle cases, compromised the reliability of downstream evaluation.

The same principle appears outside computer vision. Kapoor and Narayanan’s 2023 Patterns article on leakage and reproducibility reaches the same methodological conclusion: when multiple samples share a patient, family, recording, time block, or another dependency that matters to the prediction problem, evaluation should respect that dependency rather than assume row - level independence.

Before training, an evaluat​ion owner should be able to answer these questions:​

A good audit does not merely say that deduplication was performed. It says what relationship was tested, how it was detected, what threshold or grouping rule was used, and what remains uncertain.

​

AI dataset buyers evaluating image or video data should ask more than how many assets are available. They should ask what the provider can preserve, exclude, group, and document so the final evaluation split matches the intended generalization claim.

• Can source and derivative relationships be preserved in the delivery manifest?
• Are shoot, session, or recording identifiers available?
• Can known training assets be supplied as an exclusion list or hash inventory?
• How are exact and near - duplicate candidates detected?
• How were similarity thresholds validated?
• Can related frames, clips, people, locations, or production sequences be grouped according to our evaluation objective?
• Can the provider build or produce a fresh held - out evaluation collection?
• Will the split and exclusions be delivered under a versioned manifest?
• What limitations prevent the provider from claiming that the evaluation data are completely unseen?

For visual AI projects, those questions are often more informative than a raw asset count.

What is train-test leakage?

Train-test leakage occurs when information from the training dataset overlaps with information in the test dataset in a way that makes model evaluation unreliable. This can happen when duplicate images, related video frames, or shared source recordings appear across different dataset splits.

How can train-test leakage be prevented in image datasets?

Train-test leakage can be reduced by detecting exact and near-duplicate images, preserving source metadata, grouping related assets, and assigning entire groups to training, validation, or testing sets. The grouping method should match the model's intended generalization objective.

Can perceptual hashing detect all duplicate images?

No. Perceptual hashing can identify many visually similar images, including resized or compressed versions, but it may miss heavily edited or cropped images. It can also produce false matches. Additional methods, metadata, and manual review may be needed.

Why is train-test leakage a problem in video datasets?

Video datasets often contain many related clips and frames from the same recording. If these are randomly split across training and testing sets, the model may encounter familiar visual information during evaluation, making its performance appear stronger than it would on unseen recordings.

Does removing duplicate files guarantee a leakage-free dataset?

No. Removing exact duplicates does not eliminate all leakage risks. Related images, shared identities, source recordings, production sessions, and overlap with external pretraining data may still affect evaluation. A reliable audit must consider the intended evaluation boundary and the available evidence.

Preventing train-test leakage in image and video datasets starts with defining what the model should generalize to. Duplicate files, related images, shared source recordings, and adjacent video frames can all compromise evaluation when they cross the wrong dataset boundary.

A reliable approach combines source metadata, duplicate detection, group-aware splitting, and post-split auditing. Cryptographic hashes, perceptual hashing, image embeddings, and manual review can help identify related assets, but no single method can detect every form of leakage.

For AI teams, the goal is not simply to create unique files. It is to build an evaluation dataset that reflects the real conditions the model is expected to handle.

By preserving data lineage, documenting split decisions, and checking for cross-split overlap, teams can improve the reliability and reproducibility of their machine learning evaluations.

​​