How can I convert my custom dataset to the prefetch dataset type

Course Q&A TensorFlow: Advanced Techniques Specialization Custom and Distributed Training with TF
1

Hey mate, thanks for picking up my thread,
I am full filled with the course material, but thought the course lagged up little bit on not considering the custom dataset as their inputs, because most of the use custom dataset images for real world scenarios,

So I have a doubt, how can i convert my custom images dataset in to the prefetch dataset

Like if I have a main directory i.e images_folder

images_folder:
    |
    |_ _ _ Cats
    |           |_ _ _  cats1.jpg
    |           |_ _ _  cats2.jpg
    |           |_ _ _  cats3.jpg 
    |
    |_ _ _ Dogs
    |           |_ _ _  dogs1.jpg
    |           |_ _ _  dogs2.jpg
    |           |_ _ _  dogs3.jpg 
    |
    |_ _ _ Humans
    |           |_ _ _  Human1.jpg
    |           |_ _ _  Human2.jpg
    |           |_ _ _  Human3.jpg

How could I convert this dataset into the prefetch dataset as in the course just like this,

<PrefetchDataset shapes: (( 224, 224, 3), ()), types: (tf.uint8, tf.int64)>

Please help me, I’am eager to learn on how to do this with my own data
Help will be blessed

Thanks,
Rohit

2

Hi there, and I hope this helps with what you are doing. Here are two ways I can think of;
First of you can read your images using tf.keras.preprocessing.image_dataeset_from_directory since your images are already nicely placed in their respective label directories. Then using a mapping funtion you can apply a prefetch or interleave or whatever data optimization technique you have in mind.

Or two, you could convert your dataset into a tf.data.Dataset format and again using a mapping function apply a prefetch or any other optimization technique you desire.

In your case option one should be easier to do. Here is a link that should help guide you.

3

Thanks for picking up my question
I tried with both the thing with the gradient tape model… But it was showing rank error

4

Could you please provide the code for it because when I tried mapping and the feeding it to the model it was showing error,
Please help me,
May God bless you,
Thanks,
Rohit

5

Hi @Rohit_Kumar did you follow the link I provided, it explains everything including sample codes on how to do that. I suggest you go through the link step by step. Best of luck

6

Thanks mate, I solved it… But stummbled upon another error

I converted the the images dataset to the prefetch dataset and also got the same prefetch data as in the horses or humans assignment
But when I trained the model it was showing some shape and ranks error

Please help me with the model If any mistake made by me please correct my problems
Thanks in advance
Here is the colab link
objc_loc_personnel.ipynb (8.4 KB)

Please look into this,
This will be a great help of you @Atia

7

Hi @Rohit_Kumar could you send the notebook that contains the error messages. Thanks

8

Yeah thanks mate for helping me I have solved the error and is working perfectly fine… :+1:

9

Glad I could be of help