Certainly, I was training a model using 25 sub-volumes from one 3D image. The model learned well with that data; however, overfitting persists because I only used one 3D image. If I create all sub-volumes for 484 images, I will have 12,100 sub-volumes, and it takes too long to train the model. So, my question is, how many sub-volumes should we create for each image to achieve good results in data validation? Specifically, I want to avoid overfitting. Has anyone tried this approach before?
1 Like
How about trying total 300 images and dividing 240 training dataset(80%) and 60 validation dataset(20%)
anything lower than that can also work but will question model’s accuracy.
A total of 100 images would be must (dividing into 80-20) subset.
Regards
DP
Is creating 25 sub-volumes for each image a good choice for this dataset, or can I try making only 10 sub-volumes for each image? Please answer based on your experience. Thank you, sir.
1 Like
Experience would come with trial and error. I cannot give a definite answer on an incomplete understanding of your data and model algorithm planning.
Sub-volume can be decided based on your input shape, like in the assignment the 3D image had (240, 240, 155) which in sub-volume was converted to generate (160, 160, 16)
I hope you have gone through the data processing using patches section properly from the assignment.
No one can confirm sub-volume without knowing the data in detail. Although as I can see you are going to use less number images you could probably go ahead with 10 sub-volumes, cannot confirm.
Wait for your choice of expertise advise 
Regards
DP
You are right, the number of sub-volumes cannot be confirmed as it also depends on the model algorithm. I just don’t have much time for trial and error because the deadline for my paper is running out. Thank you, sir, for the advice.
1 Like
1 Like
Hello @Elvaret_M168DSX1546
your error is coming with the line mentioned for X[i, :, :, :]
Can I know the reason behind transposing both x data and y data here??
Based on this webpage (3D Data-Augmentation Using tf.data and Volumentations-3D Library | by Fakrul Islam Tushar | Medium) and this repository (volumentations/example.py at master · ashawkey/volumentations · GitHub, the Volumentations library for augmentation requires the input shape to be in the form [imgs: [D, H, W, C], where the depth or batch dimension should be the first]. To match this shape, I transpose the dimensions (because my X dimension is [num_channel, dim_x, dim_y, dim_z]), and after the augmentation, I revert the shape back.
1 Like
Hello Elvaret,
the reason I think this issue is arising because you are mask for the y data after the transpose of x data and y data.
Can I know reason behind using mask for the y data??