I have a question about weight matrices in simple dense layer.

Since the matrices are general, when the input/output dimensions are quite high, the number of parameters becomes huge.

What if we could restrict to particular types of matrices? This way, the dimensions of the matrix stay the same, but the number of parameters can be way smaller.

For example, rotation matrices in some dimensions, or combinations of them?

More generally, what if we restrict the form of matrices to certain matrix group representation? Would certain groups be better for particular layers or tasks in general?

I’ve made a few experiments with SO(2) and SO(3) groups, and wrote about it here: