Why quantisation using Quanto from HF only for Linear Layers?

Short Course Q&A Quantization Fundamentals with Hugging Face
1

Hello Community,

I’ve been experimenting with Hugging Face’s Quanto library for quantization and noticed that it currently applies quantization only to Linear layers. This got me thinking:

Why are Linear layers the main focus?
Is it because they dominate the parameter count in transformer-based models? Or is it due to implementation simplicity and hardware support for matrix multiplications?

What about other layer types?
For example, quantizing Embedding layers or LayerNorm could also help with memory footprint. Is it that quantizing these layers gives less benefit or introduces too much accuracy degradation?

Thanks in advance.

2

Perhaps this is because the reconstruction from quantization or the loss of information can be more tolerated in Linear layers since they have a more predictable relationship.