General question about weight matrices in Dense Layer

Course Q&A Deep Learning Specialization Neural Networks and Deep Learning
, , , ,
1

I have a question about weight matrices in simple dense layer.
Since the matrices are general, when the input/output dimensions are quite high, the number of parameters becomes huge.
What if we could restrict to particular types of matrices? This way, the dimensions of the matrix stay the same, but the number of parameters can be way smaller.
For example, rotation matrices in some dimensions, or combinations of them?
More generally, what if we restrict the form of matrices to certain matrix group representation? Would certain groups be better for particular layers or tasks in general?

I’ve made a few experiments with SO(2) and SO(3) groups, and wrote about it here:

2

You should not force any assumptions on what each layer is going to learn. The optimization will adjust the weights to minimize the cost.

3

Sure, but isn’t the CONV layer or any other specific layer exactly forcing particular assumptions?

4

That just specifies a convolution - it doesn’t constrain what the convolution weights will learn.