Could someone explain the intuition behind replacing the chosen hidden tokens with either MASK, a random token, or original token in the pretraining step? Why are all three used?

BERT pretraining

arvyzukai February 6, 2024, 6:11pm 2

Here is my previous attempt at explaining this.

Cheers

Topic		Replies	Views
Transformer Decoder Mask Input NLP with Attention Models week-module-3	1	521	August 12, 2022
Few doubts regarding the pre-training and working of t5 transformers NLP with Attention Models week-module-3	2	333	November 9, 2023
Confusion regarding the video on BERT Objective NLP with Attention Models week-module-3	2	367	September 4, 2023
Understanding Masking NLP with Attention Models week-module-3	3	561	September 19, 2023
Predicting Next Set of Tokens in Decoder Model Generative AI with Large Language Models week-module-1	7	578	August 10, 2023