At the end of the assignment, once you’ve completed the colab notebook, it says:
Then, if you’re brave enough, you can edit the Image Detection app for Rock, Paper and Scissors only, instead of the 1,000 classes it could recognize!
But we are not taught anywhere how to do this
Does anyone know what things we must change in order for the app to work with our new model and labels files?
Has anyone ever been able to actually complete this task?
Apart from changing
what else must we change in the app in order for it to work?
I’m currently getting no message stating what has been classified
My problem seems to be here:
The error is:
Cannot copy to a TensorFlowLite tensor (serving_default_keras_layer_input:0) with 602112 bytes from a Java Buffer with 150528 bytes.
From reading around I think the problem may be an incompatibility with the mobilenet_v2 used to generate our converted_model.tflite file and the mobilenet_v1 model that was used when writing the kotlin image classifier code (v1 uses a different data type in its input).
However, I don’t know how to fix this
Any ideas?
Hi Jaime, thanks for your question.
The optional assignment for object detection in android only asks you to use an existing object detector(MobileNet to be precise) that was previously trained on the ImageNet dataset. It’s a bit complicated, but I will try to explain.
Most object detectors that are provided in TF Lite were previously trained(or pre-trained) on the ImageNet dataset, a popular dataset that contains 1000 object categories. If you use those detectors to detect certain objects(like rock, paper scissors) without changing anything about the detector(running a model directly on test samples is normally called performing inference), the results take all 1000 classes into account, and in the worst case, if the objects you are trying to detect are not available in ImageNet 1000 categories, your detector will miss those objects(or detect them but assign them a wrong category).
I think this is what you are doing in the assignment. You are merely using a pre-trained Mobilenet to recognize objects. If you want to detect your own or custom objects, you have to change something. To detect custom objects, we typically fine-tune the pre-trained detectors on our own dataset that contain a fixed number of categories(3 in the brave case) - this is normally called transfer learning & fine-tuning or pre-training, ML terms are always ambiguous but you get the idea). Transfer learning is most known in image classification but it also works in object detection, and overall, the process is quite similar.
So, as the assignment said, if you are brave enough, you can make your custom MobileNet detector. You will take an existing MobileNet detector and customize it on your dataset so rather than detecting all ImageNet 1000 classes, it only detects 3 fixed classes(rock, paper, scissors).
If you did like to do this, you can take look at TensorFlow Lite documentation. There are steps and guides that can help you make a custom rock, paper, and scissors detector. Since you are doing this in Android, it might take you some time to get it to work but I believe it can work. You can also use TF Lite Model Maker, “a transfer learning library to train custom TFLite models”(seems it can be much easier).
Note: There is a note on TF Lite doc that says “the pre-trained models we provide are trained to detect 90 classes of objects” but generally all detection models that are available in TensorFlow Hub were trained on ImageNet 1000 object classes.
Hope that helps!
Hi @Jean_de and @Khushwanth
I have managed to make it work!
As I suspected, the model we build in the C2W2 colab has float inputs (image below from netron.app, great tool to visualise models):
But the model used in the Image Classification android studio app provided has uint8 inputs:
This is why when I substituted the new model we created in the colab (called ‘converted_model.tflite’) with the model provided in the IImage Classification android studio app (called ‘mobilenet_v1_1.0_224_quant.tflite’) I got an error (‘Cannot copy to a TensorFlowLite tensor (serving_default_keras_layer_input:0) with 602112 bytes from a Java Buffer with 150528 bytes’)
So after lots of reading around of stack overflow and tensorflow docs and banging my head against the wall I went back to the Colab (C2W2) to alter the process where we convert the model to tflite.
To be more specific, I made sure that in the conversion process, when we quantize, both the input and output of the model were converted to the uint8 datatype.
This is known as “integer-only quantization”, and here is the code to do it from the Tensorflow docs:
def representative_data_gen():
for input_value in tf.data.Dataset.from_tensor_slices(train_images).batch(1).take(100):
yield [input_value]
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_data_gen
# Ensure that if any ops can't be quantized, the converter throws an error
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
# Set the input and output tensors to uint8 (APIs added in r2.3)
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
tflite_model_quant = converter.convert()
But in order for that code to work in C2W2, it needs to look like this:
def format_image(image, label):
image = tf.image.resize(image, IMAGE_SIZE) / 255.0
return image
BATCH_SIZE = 32
train_images = train_examples.shuffle(num_examples // 4).batch(BATCH_SIZE).map(format_image).prefetch(1)
def representative_data_gen():
for input_value in train_images.take(100):
yield [input_value]
converter = tf.lite.TFLiteConverter.from_saved_model(RPS_SAVED_MODEL)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_data_gen
# Ensure that if any ops can't be quantized, the converter throws an error
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
# Set the input and output tensors to uint8 (APIs added in r2.3)
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
tflite_model_quant = converter.convert()
Downloading and saving this other model in the Image Classification android studio app now works
I am not a mentor in the D&D specialization, but since you guys are, I would suggest posting this discovery and solution as a gitissue, it may help future learners if this is fixed
Jaime
For future learners: