In BRNN, is backprop and forward prop done simultaneously?
If not then how is it carried out?
1 Like
Yeah they are done both at the same time.
2 Likes
Only after the entire sequence is processed from both directions is the output from BRNN made avaiable. As far as invoking the child RNNs are concerned, keras implementation does it sequentially. In theory, this can be done in parallel.
BRNN uses 1 RNN for processing the input from each end (totals to 2). When making predictions, outputs of both these child RNNs are combined to generate output at every timestep. In tensorflow, if you provide just 1 layer as input (i.e. no backward_layer) to a BRNN, a clone of the forward direction layer will be used for learning from the backward direction. Do look at the merge_mode flag to understand how outputs are returned.
As far as backpropagation is concerned, in theory, the child RNNs can update in parallel.
2 Likes
So it doesn’t surprise me that backward propogation and forward propogation are tied together, because gradient descent can’t happen without backward propogation. Backward propogation literally just means taking the derivative.
But the bi-directional part of the BRNN refers to the way the graph of the units is connected. In principle, if back propogation takes place in a manner that extends across all the units rather than independently within a unit, it should proceed from the “last unit” to the “first unit”. If the BRNN is aligned left to right then right to left, isn’t the “last unit” the left most, so that back prop would go to the right, then back to the left?
Backprop for each step will be influenced by both directions.
As far as the implementation is concerned, auto gradient implementations are better left to it.
1 Like