05_Training_lab_student tokenize_and_split_data fuction

Short Course Q&A Finetuning Large Language Models
1

in 05_Training_lab_student, there is a function call:
train_dataset, test_dataset = tokenize_and_split_data(training_config, tokenizer)
how can I see inside the “tokenize_and_split_data” function and implement it by myself?

1 Like
2

There are many ways to look at source code of a function.
Use either the inspect module or ?? in a jupyter notebook to view function definition.
Example below shows how to look at the source code of encode function inside base64 module.

image
image798×972 67.1 KB

3 Likes
Where is the utils package?
3

thanks for your guidance
I also found this solution helpful:

%load utilities

this way helped me to see all functions and variables that were defined in utilities package

1 Like