This may be a little bit outside the course material but the implimentation of dropout made me think of this.
I am interested in optimizing a hemispheral neural network where inputs are modeled in different “hemispheres” based on topic assigned by the researcher and are then combined in the final layer, as outlined in Coulombe 2022. What would be the best way to implement forward- and back propagation for this framework? One approach would be to use different inputs (X’s) for each hemisphere and implement back propagation separately (perhaps using a for loop for each hemisphere). Another thought was that I could zero out weights for inputs in different hemisphere, essentially vectorizing the implementation. Any thoughts on the costs and benefits of either approach?
Hi @Will_Ensor,
I have not read the text but I think you can implement the architecture shown in the graph on page 5 as one single Tensorflow model that accepts 3 inputs and produces 1 output. If you check out this link, you will find the following example that clearly shows that there are 3 InputLayer that go through different paths (in your case, different hemispheres) but all paths get concatenated by the Concatenate layer and finally produces two output layers.
You only need to replace those paths with your hemispheres, and keep only one output layer that does the summation (shown in the graph on page 5) after Concatenate.
Note that I am not analyzing nor comparing the benefits between your suggestions and mine, because I think it will be for you to figure out yourself through experiments and with a good understanding of the paper’s method.
Cheers,
Raymond
Thank you, this is helpful! Will have to dig into this is and see what keras is doing under the hood.
