What context direction of Encoder-Decoder AKA Seq-to-Seq model

Course Q&A Generative AI with Large Language Models
1

In this lecture,

for Encoder Only Context is Bi-directional
for Decoder Only Context is Uni-directional
but Encoder-Decoder context was missed in the lecture

Can you help with context direction for Encoder-Decoder model

2

Hi @Naren_babu_R ,

I think I can help you with your question.

As you state:

  • Encoder-Only will look at context bi-directionally
  • Decoder-Only will look at context in one direction only, and will predict the next token.

Then we have Encoder-Decoder models. In this case you have both of the above cases.

This model has the 2 components. The initial input starts at the encoder, where the model can see the entire context (bi-directional). Once the encoder delivers its output to the decoder part, at that moment, in the decoder, the model will only see the context in one direction.

So the encoder-decoder models have both “behaviors”, depending on where the process is at: the encoder part or the decoder part.

Does it make sense?

3

yes. thanks