Reduce training time

AI Discussions
1

I wish to train a CNN on radiography images. My training set contains around 500,000 images and there are 22 classes with each image having one or more classes as positive. I’m using Colab Pro to train the model and the dataset is located in my Google drive. The time it takes for the first epoch is around 12 hours. How can I reduce this training time? I believe the location of my dataset (google drive) is one of the major factors that increases the I/O time. Are there any solutions to reduce that? I’ve heard that using using google cloud platform to store my data will reduce the I/O time while reading data but I’m not sure if this is true. Kindly share some thoughts.

1 Like
2

Hello @Harshit1097! I hope you are doing well.

Yes, you are correct, the location dataset (Google Drive) could be one of the factors that increase the I/O time. Using Google Cloud Storage to store your data will allow accessing data more quickly but I don’t know how quickly.

However, another approach to consider is transfer learning. I think it would be the best solution to your problem. Check this video for more about transfer learning.

Best,
Saif.

1 Like
3

Hello @saifkhanengr ! Yes I’m using EfficientNet model pretrained on ImageNet dataset for my purpose.

4

Sounds good. Then share your experience and insights with us.
Good luck with your research!

Best,
Saif.

5

[2110.06512] MedNet: Pre-trained Convolutional Neural Network Model for the Medical Imaging Tasks.

“multiple classification methods from medical imaging make use of TL from general-purpose pre-trained models, e.g., ImageNet, which has been proven to be ineffective due to the mismatch between the features learned from natural images (ImageNet) and those more specific from medical images especially medical gray images such as X-rays. ImageNet does not have grayscale images such as MRI, CT, and X-ray.“

Emphasis added

6

I may be completely mistaken, but my understanding is that you have a model pretrained on ImageNet, and are now trying to retrain some/all of it using 500,000 medical images.

My thoughts:

  1. as indicated above, it’s likely counterproductive to use a model trained on color, non-medical images to train a CNN that will operate on black and white medical images. Even if the medical images are color, say dermatology / skin photos, it is questionable whether the features are sufficiently alike.

  2. do you have empirical evidence suggesting that you actually need 500,000 training inputs to get good results? 400,000 wasn’t adequate? Or 50,000? Or 15,000? I would run experiments on datasets 10 and even 100x smaller to see how much data is really needed/impactful.

  3. in some recent threads about minimum training data requirements, mentor @Christian_Simonis has provided links to papers or articles suggesting as few as 10 new training inputs can yield good results in certain transfer learning environments. If you haven’t looked at those yet, you should do. I haven’t done the experiment myself but just have a hunch that using 10 inputs instead of 500,000 might go a long way towards your goal of reducing training time :grinning:

Ps: if you didn’t know, you can find a user’s digital exhaust by clicking on their @name, clicking to expand their profile, and selecting Activity. Cheers

1 Like