After quantization the model (running the dequantize_model function) initial model does not make inference (RuntimeError: “LayerNormKernelImpl” not implemented for ‘Byte’). After the dequantization it works again (but the output of this initial model is same as for quantized / dequantized model). So I suppose that we change initial model during these steps, maybe we need to use a copy of a model if we will need to use it again for comparison (on load it again).
Could you explain this behavior?
Also is it reasonable to make a quantization not for all layers? For example 50% of layers are quantized, other left as is