How do I implement a sparse locally connected neural network

AI Discussions
, , ,
1

I’m trying to build a image based neural network in tensorflow where the output shape is the same as the input shape, and I’m trying to make a sparsely connected neural network. The sparse connections should not be randomly connected (for example using Dense and Dropout),

rather I want to build a model with layers in which the nodes are in a 2d arrangement, so that the nodes in a layer are connected to only the nearest n nodes in the next layer (for example, giving a kernel size of 5 would mean that a node in a layer is connected to 25 (5x5) nodes in the next layer.

This is because features from the bottom right of the image should have no connection to the features in the top left of the image.

The main reason to do this is to reduce computational load. (If I only use dense layers, for a 512 x 512 image, for both the input and output images, that would mean 262144 nodes per layer and that would be too many parameters which are unnecessary)

2

Hi there, I am not an expert in this, but if tensorflow or torch don’t have these options, then you gotta build from scratch your own neurons and layers, I think…

1 Like
3

What problem are you solving for such a custom NN instead of using tfmot or Conv / Pool layers?

5

It would be unfortunate if tensorflow and torch don’t have these built in, because I think training would be unoptimized and slow in a model built from scratch

6

Have you tried googling “TensorFlow sparse”? Here’s one of the links I get.

I have never worked with those, so have no personal experience to contribute. But a quick scan of the first few paragraphs shows that those may not be relevant to what you are trying to do: they are for the case in which you know that your matrices are sparse. It sounds like you are trying to force sparsity, as opposed to deal with it when it occurs.

But maybe if you read all the relevant docs here, you’ll learn something that would be helpful …

8

I’m actually trying to take 2 stereo images as a left branch and right branch of neural network layers, then concatenate them, and add a few more layers and finally output a depth map.

So the input would be 2 input images with a lateral shift between them and the output would be a depth map (either a 3d point cloud or a 2d depth map).

I thought of using Conv2d and Conv2dTranspose, to generate the images but I thought the concept of a sparsely connected neural network would also work well, but it’s hard to find a way to implement it

9

I have tried using those, but the problem is the inefficiency. 512 x 512 is 262144 nodes in a layer and for each node it would have to hold 512 x 512 indices in a sparse matrix, so it would be 68719476736 numbers stored just for the indices of sparsity between 2 layers.
and I want the model to have multiple layers.

The sparse matrices can be generated for each node at the time at which it is needed, but I don’t know how this would affect the training and training times