Does Number of Fully connected neural networks changes in transformer architechture based on max length input size?

Course Q&A Deep Learning Specialization Sequence Models
1

Considering the architecture of encoder and decoder in transformer as shown below:

image
image1082×694 28.5 KB

  • Does each input token after self attention mechanism (z1,z2,z3,…)is passed to it’s specific separate Feed forward neural network or does all the Z’s are stacked into one and then passed to single FFNN?
  • If all the Z’s are stacked into one, then how the difference in shapes of different inputs is taken care
  • If every z has its own Feed forward neural network, how in practical it is implemented with arbitrary input length?
2

Hi Arjun_Reddy,

  • the z’s are stacked and then passed to a single feed forward layer
  • the difference in shapes of inputs is resolved by padding to the dimension of the model

Hope this clarifies.

1 Like