Fine-tuning and chinchilla paper

Course Q&A Generative AI with Large Language Models

Derar_Saifan November 18, 2023, 9:10am 1

Hi,

Does the chinchilla paper guidelines of having a number of token =~20 times the number of parameters in pre-training apply for instruction fine tuning? Any recommendation on the size of the instruction fine tuning data size?

Thanks

Topic		Replies	Views
Optimal Number of Tokens Generative AI with Large Language Models week-1	5	509	February 9, 2024
Does BloombergGPT contradict Chinchilla and Llama papers? Generative AI with Large Language Models week-1	4	514	July 7, 2023
Right-Sizing Models for the Dataset: Finding the Best Data-To-Parameter Ratio for NLP Models AI Discussions the-batch , ai-discussions	1	71	May 20, 2023
Week 2: Instruction fine-tuning Generative AI with Large Language Models llm , prompting	1	57	November 18, 2024
On Scaling Laws and Compute-Optimal Models lecture Generative AI with Large Language Models week-1	2	444	June 30, 2023

Fine-tuning and chinchilla paper

Related topics