How to extract body of a transformer like models and fine tune with that body on different shape dataset

Course Q&A Deep Learning Specialization Convolutional Neural Networks
1

In BERT like transformer model, it has 2 training objectives Masked Language Modeling and Next sentence prediction right? and BERT model is also supports different input shapes, So I am actually building a model with 2 training objectives on a base model and want to take the base model body and fine tune the model on different data with different shape in TensorFlow, How is code written for this in tensorflow at low-level.

2

Dear @Arjun_Reddy ,
To fine-tune a BERT model with two training objectives on a base model and different data with different shapes in TensorFlow, you can follow the steps outlined in the TensorFlow documentation. First, you can download the configuration file for the pre-trained BERT model and build the model using the TensorFlow Model Garden. Then, you can use the KerasLayer from TensorFlow Hub to wrap the input preprocessor and the model. Next, you can define the loss function, optimizer, and metrics for the model and compile it. Finally, you can train the model on your data using the fit method. You can also use transfer learning and fine-tuning to adapt the pre-trained BERT model to your specific task. TensorFlow provides a complete guide to transfer learning and fine-tuning in Keras, which includes creating a new model on top of the output of one or several layers from the base model, freezing the base model, and training the new model on your data.
Best,
Can

3

Thanks for sharing, But I want to know whats happening internally, like how is this ambuiguity is been able to capture in form of code, Like I am currently building a model with multiple inputs and multiple outputs but stuck at place, Where I don’t know how to remove inputs and outputs and just using base model for fine-tuning in tensorflow, I want help in this.