Hi all,
While working through the Siamese network module, I think I found a data-leakage issue in how the train/validation split is constructed for SignatureTripletDataset, and wanted to flag it in case it’s useful.
The setup:
SignatureTripletDataset.__getitem__(self, index) ignores the index argument entirely. It generates a triplet by calling random.choice(self.user_ids) and then randomly sampling images for that user, regardless of what index was requested.
The issue:
create_signature_datasets_splits() creates the train/val split using random_split(full_dataset, [train_size, val_size]), which partitions indices, then wraps each resulting Subset in a TransformedSignatureSubset.
The __getitem__ method of the subset then simply calls the __getitem__ of the underlying full dataset.
But since __getitem__ never actually uses the index it’s given, this split has no effect on what gets sampled and both the “train” and “val” subsets end up drawing triplets from the exact same pool of all users/images in full_signature_dataset.
In practice, this means the same individuals (and potentially the very same images) can appear in both the training loop and the validation loop every epoch, so the validation metric isn’t measuring generalization to held-out data but is just measuring performance on a different random sample from the same population the model is training on.
Note this is separate from the 51-vs-1 individual held out for final testing and the leakage is specifically in the 80/20 train/val split within the 51 profiles used for training.
Possible fix:
Split at the user-ID level before any subset construction (e.g., shuffle user_ids, assign disjoint subsets to train/val), and have SignatureTripletDataset accept a restricted list of users so random.choice(self.user_ids) only ever draws from the users assigned to that particular split. That would guarantee no individual appears in both train and val.
Just wanted to raise it in case others hit the same issue or in case I’m missing something about how the split is intended to work.
Thank you!