How CodeLLMs are trained

AI Discussions
1

Hello

  1. How do we train a code generating model. Are we just doing it Auto regressively or are there any specific ways to do so. Suppose I have an LLM , how do I fine tune it to generate codes. So will I follow next token prediction or what?

  2. How the model is able to generalize the code. Based on the question we ask it generates code. But it is a model which generates next token based on probability. But there are potential chances that the syntax may slightly get incorrected due to selecting wrong word. I mean how the model overcomes syntax, runtime and other errors.

2

Have you searched for academic papers on the topic? That sort of work is state-of-the-art, and there is lots of commercial activity (OpenAI, Meta, Google, etc).

Iā€™m not sure to what extent their methods are proprietary or public domain.

I suspect a lot of the keys to this technology are not widely discussed in public.

1 Like