Hi @PZ2004
I don’t quite understand you question. Please read the last paragraph of this post and tell if you still have questions. In short, in T5:
Self-supervised training uses corrupted tokens, by randomly removing 15% of the tokens and replacing them with individual sentinel tokens (if several consecutive tokens are marked for removal, the whole group is replaced with a single sentinel token). The input of the encoder is the corrupted sentence, the input of the decoder is the original sentence and the target is then the dropped out tokens delimited by their sentinel tokens.
While in our assignment we do not implement the decoder part (and also other T5 tasks) so the assignment’s encoder has to predict the sentinels correctly.
Cheers
P.S. use “\” to escape the “<” symbols (that what makes your text bold or striked out and it is hard to read).