Preventing Train – Test Leakage in Professional Image and Video Libraries

Data leakage in machine learning happens when information that should be unavailable during model development influences training, preprocessing, model selection, or evaluation. It can make validation or test performance look stronger than real - world performance because the evaluation boundary has already been compromised.

For teams building models with AI training datasets, one of the easiest forms of leakage to miss is train - test contamination through related visual content. Different file names and asset IDs can still represent the same image at another resolution, a crop or edit of the same source, adjacent frames from one recording, or observations that share an identity or production session that was supposed to remain unseen.

The practical issue is independence. A test split is only useful when it measures the kind of generalization the system will need after deployment, rather than rewarding the model for seeing a closely related version of the same underlying source during training.

An unseen asset ID does not necessarily mean an unseen visual source.

Data leakage occurs when information that should remain outside model development affects the trained model or the decisions used to build it. The result is usually an evaluation score that does not represent the intended deployment setting. In this article, “data leakage” refers to a machine - learning evaluation problem, not the cybersecurity meaning of unauthorized data disclosure.

Train - test contamination occurs when training information crosses into validation or test data, or vice versa. Exact duplicate records are the simplest example, but the same problem can arise when nominally different samples share a source, identity, session, sequence, or other dependency that the evaluation was supposed to hold out.

Preprocessing leakage occurs when a transformation is learned using information from data that should have remained held out. Common examples include fitting normalization parameters, imputers, feature selection, or other learned preprocessing on the complete dataset before evaluation instead of fitting them on training data only.

Target leakage occurs when model inputs contain information about the outcome that would not legitimately be available when the prediction is made.

The main focus here is train - test contamination in image and video datasets: keeping evaluation data genuinely independent from training data while preserving the relationships needed to audit that boundary.

Data leakage is also different from ordinary overfitting. A model can overfit a perfectly clean training set. Leakage compromises the evaluation boundary itself, so the reported test metric can become an unreliable estimate of generalization. For a broader discussion of this problem in scientific machine learning, see Kapoor and Narayanan’s Leakage and the reproducibility crisis in machine learning based science.

​

This is especially important in computer vision datasets, where file - level uniqueness does not guarantee independent observations. Images and videos can be strongly related even when every file name is different.

A professional image library can contain one photograph at several resolutions, edited and unedited derivatives, photographs taken a fraction of a second apart, or many angles of the same person or product from one shoot.

Video adds more relationships. One recording may be delivered as several trims, aspect ratios, codecs, resolutions, or short clips. A frame - extraction pipeline can then turn one recording into thousands of individual images.

These records can all have legitimate reasons to exist separately in a media catalog. That does not mean they should automatically be treated as independent observations in a machine - learning experiment.

For evaluation design, the relevant question is whether a relationship between assets makes the test easier than the deployment problem the experiment claims to measure.

A clean split depends on what "unseen" is supposed to mean

There is no universally correct grouping key for visual datasets.

A random split is reasonable when samples can genuinely be treated as independent observations from the target distribution. Problems start when the data contain a dependency that matters to the evaluation claim.

Choose the split unit from the generalization claim first, then randomize at that unit if randomization is appropriate.

Group - aware validation tools implement the same principle computationally. For example, scikit - learn’s GroupShuffleSplit lets samples share a group label so that a group is held out as a unit when creating train - test partitions.

It is useful to distinguish three problems because they require different evidence.

1. Exact duplicates

These are byte - identical files. A cryptographic hash is usually sufficient. If two files produce the same strong content hash, they can be treated as identical for ordinary dataset - deduplication purposes. This check is cheap and should generally be part of ingestion or dataset QA. It is also the easiest case.

2. Near - duplicate visual variants

These are substantially the same observation after transformations such as recompression, resizing, modest cropping, color adjustment, rotation, or another edit. Byte hashing will not find them because the underlying bytes have changed. Perceptual hashing can be useful here because it encodes aspects of image structure into a compact signature that can be compared using Hamming distance. It should not be treated as a perfect duplicate oracle. Local edits, larger crops, overlays, or more substantial transformations can produce false negatives, while a loose distance threshold can group distinct images. Thresholds should be tested against representative pairs from the actual collection.

3. Related or dependent samples

The hardest category is not necessarily duplicate content at all. Two photographs may show the same person from different viewpoints. Two clips may come from the same recording but contain different moments. Two photographs of a product may be visually distinct while sharing the same controlled studio setup. Embedding retrieval can help surface these relationships because learned representations can retrieve visually or semantically related assets that simple hashes miss. But similarity is only evidence. A high cosine - similarity score does not by itself prove that two records belong in one leakage group. The correct decision depends on the embedding model, threshold, available metadata, source lineage, and, most importantly, the evaluation objective.

Derivative assets cross the split

A source photograph may exist as a full - resolution file, resized delivery, edited version, crop, or re - export.

Video creates even more derivative relationships: multiple encodes, vertical and horizontal crops, extracts, previews, trimmed versions, and frame sequences.

If derivatives receive independent asset IDs and the split is performed at the ID level, different representations of one source can appear in training and testing.

Capture sessions cross the split

A photoshoot can produce tens or hundreds of related frames.

Whether they must remain together depends on the task. If the evaluation is supposed to measure generalization to new sessions, then randomly distributing frames from one session creates an unrealistically easy test. The same applies to repeated takes from one controlled video shoot.

Adjacent video frames cross the split

Video frames are temporally correlated by construction. If frames are extracted densely and then shuffled independently, frames separated by a fraction of a second can land in different partitions. For frame - level datasets, preserve the source video, sequence position, and timestamp instead of treating every extracted image as an unrelated example.

Related data arrives through multiple collections

Large media libraries are often assembled over time. The same underlying content may arrive through separate acquisitions or exist in both an original archive and a later derivative collection. If source lineage is incomplete, a per - collection deduplication pass may not discover the cross - collection relationship.

Visual retrieval can therefore be useful across the full candidate corpus, not just within one supplier or folder.

Public benchmarks overlap training corpora

A separate problem arises when evaluation assets are publicly available and a model has been pretrained on large external corpora.

A 2025 ICCV Workshops study, Data Leakage in Visual Datasets, examined overlap within and between visual datasets and used image retrieval to show that all analyzed datasets presented some form of leakage; the authors also found that leakage compromised downstream evaluation reliability.

This kind of contamination is harder to rule out when the evaluator has no access to a model's pretraining corpus. A clean fine - tuning split therefore does not prove that a public benchmark was unseen during pretraining.

​

A defensible workflow starts by defining what the test set is supposed to be independent from, then preserving and auditing those relationships before and after the final split.

1. Define the evaluation claim

Write down what the model is expected to generalize to: new files, recording sessions, people, locations, cameras, time periods, or another factor. That decision determines which relationships must remain isolated.

2. Preserve source lineage

Use existing production information before trying to reconstruct every relationship with computer vision. Useful fields can include original source ID, derivative - of ID, shoot or session ID, recording ID, contributor or acquisition source, participant or subject ID where appropriate and lawfully available, timestamps and frame position, and edit or export relationships. Lineage is often stronger evidence of dependency than visual similarity alone.

3. Fingerprint the collection

Use a staged audit. Start with cryptographic hashes for exact identity. Add perceptual hashes where transformed duplicates are plausible. Use embedding retrieval where near - duplicate or related - content search is needed. At large scale, an approximate nearest - neighbor index can retrieve candidate neighbors without exhaustively comparing every possible pair. Treat similarity as a screening signal, not an automatic grouping decision.

4. Build groups conservatively

Do not automatically merge every asset connected by any similarity edge. If image A is similar to B and B is similar to C, that does not prove that A and C should share one evaluation group. Decide which relationships are legitimately transitive. Combine source metadata, visual evidence, and manual review for ambiguous cases. Inspect unusually large clusters because they can reveal an overly permissive threshold or semantic chaining.

5. Assign groups, not individual records

Once the relevant groups have been defined, assign whole groups to training, validation, or testing. The group might be an exact - duplicate family, source recording, shoot, participant, location, or another task - specific unit. Randomization can still be used; it simply happens at the appropriate unit.

6. Keep learned preprocessing inside the training boundary

Grouping and fixed duplicate detection before split assignment should not be confused with fitting a model transformation on the entire dataset. Once the evaluation partitions are established, preprocessing that learns parameters from data should be fitted using training data only and then applied to validation or test data as appropriate. The protected test set should not become an iterative model - development tool.

7. Freeze and version the evaluation set

A split should be reproducible. Keep a versioned manifest containing stable asset identifiers, group identifiers, exclusion records, reference labels, and the split assignment. If the library changes, do not silently change the benchmark. Decide whether a new acquisition belongs in training, a future evaluation refresh, or neither.

8. Audit the boundary after splitting

After the split has been created, query test assets against the training index. Inspect the strongest cross - split matches and record why each candidate is or is not considered problematic. The audit is evidence that the split was checked; a similarity score by itself is not.

Useful audit outputs include:

• exact cross - split matches

• near - duplicate candidates

• shared - source relationships

• shared identities where identity isolation is required

• sequence or frame adjacency

• unexpected cluster sizes

• manual - review results

• unresolved limitations

The practical workflow is usually a combination rather than a competition between methods. A byte hash is cheap enough to use broadly. Perceptual hashing provides another inexpensive candidate layer. Embedding retrieval can search relationships that those methods miss. Source metadata can then confirm relationships that pixels alone cannot establish.

​

Video datasets for AI training need additional controls because a single source recording can produce many highly correlated clips or thousands of extracted frames.

At minimum, preserve a hierarchy such as: production/session  - > source recording  - > derived clip  - > extracted frame.

Then decide which level is the evaluation boundary. If several clips come from the same recording and the objective is generalization to unseen recordings, all of those clips should stay together.

If frames are extracted, retain timestamps and source IDs. Do not split densely sampled frames independently and then interpret the held - out frames as genuinely new video observations.

Recurring people or locations require a separate decision. If the model is expected to generalize to unseen individuals, identity is a grouping factor. If the deployment product explicitly operates on a known group of people, identity overlap may be appropriate. Again, the generalization claim determines the group.

​

The consequences are not hypothetical. In the CVPR 2026 paper Data Leakage Detection and De - duplication in Large Scale Geospatial Image Datasets, researchers studying the AICrowd Mapping Challenge dataset reported that nearly 90% (about 250,000) of training images had identical copies and that approximately 56,000 of 60,000 validation images were also present in training, amounting to 93% train - validation leakage.

That is an extreme case, not a general estimate for image datasets. Its value is in showing how a benchmark can look structurally plausible while containing enough redundancy to make evaluation results misleading.

More broadly, the 2025 Data Leakage in Visual Datasets study found leakage in all of the visual datasets it analyzed and reported that leakage, from severe to subtle cases, compromised the reliability of downstream evaluation.

The same principle appears outside computer vision. Kapoor and Narayanan’s 2023 Patterns article on leakage and reproducibility reaches the same methodological conclusion: when multiple samples share a patient, family, recording, time block, or another dependency that matters to the prediction problem, evaluation should respect that dependency rather than assume row - level independence.

Before training, an evaluat​ion owner should be able to answer these questions:​

A good audit does not merely say that deduplication was performed. It says what relationship was tested, how it was detected, what threshold or grouping rule was used, and what remains uncertain.

​

AI dataset buyers evaluating image or video data should ask more than how many assets are available. They should ask what the provider can preserve, exclude, group, and document so the final evaluation split matches the intended generalization claim.

• Can source and derivative relationships be preserved in the delivery manifest?
• Are shoot, session, or recording identifiers available?
• Can known training assets be supplied as an exclusion list or hash inventory?
• How are exact and near - duplicate candidates detected?
• How were similarity thresholds validated?
• Can related frames, clips, people, locations, or production sequences be grouped according to our evaluation objective?
• Can the provider build or produce a fresh held - out evaluation collection?
• Will the split and exclusions be delivered under a versioned manifest?
• What limitations prevent the provider from claiming that the evaluation data are completely unseen?

For visual AI projects, those questions are often more informative than a raw asset count.

Train - test leakage is not fundamentally a file - management problem. It is an evaluation - design problem: the split must protect the relationship the model is supposed to generalize beyond.

A dataset can contain unique file names, unique IDs, and no byte - identical duplicates while still producing an unrealistically easy test. Conversely, two visually similar samples do not automatically constitute leakage if their shared information is part of the intended deployment distribution.

The useful question is therefore not simply whether two assets look alike. It is whether information crosses an evaluation boundary that the model is supposed to prove it can generalize across.

Define that boundary first. Preserve source lineage. Use hashes and retrieval methods to find relationships that metadata misses. Group samples according to the actual evaluation claim. Keep learned preprocessing and model selection inside the training boundary. Then freeze, version, and audit the resulting test set.

A clean split is a claim about independence, not about file IDs.

​