How can I fine-tune a lightweight LLM specifically for JavaScript frameworks?

AI Discussions
,
1

I am a technical architect for many years now but I am very new to Machine Learning. I have a requirement and want to have some help from the community.

I want to fine-tune a model on the JS stack. Currently, all big LLMs are general purpose and have all the things. But what I want is to have a custom model trained only for the Javascript stack and that too not all the JS ecosystem but the specific frameworks that I use day to day. The idea is to have a tiny model that can run on my Macbook and is strictly just about coding. Following are the things which I am looking for in it

  1. Code completion

  2. Explain the code

  3. Document the code (optional)

  4. Review code and suggest optimizations (based on best practices)

  5. Finding issues and bug fixing

  6. Ability to mention the framework and get answers strictly based on that for e.g. if I say @express then it should answer only based on the express js framework.

  7. Ability to chat and ask coding questions.

  8. Understand my code base and give answers on its basis.

I know I have mentioned a lot here but this is the roadmap that I wish to follow. I want it to be all open source without paying to any big guns. Of course, I will open-source this also.

I am seeking help from the community to guide me in the right direction.

  1. Which base model should I pick (this mode is going to be strictly for coding, so I don’t want all the other stuff that is baked in those big LLMs)? It should be the lightest possible which can be fine-tuned and do the job.

  2. Should I choose just a base model or instruct ready as I want to be able to chat with it also?

  3. What is the best way to fine-tune the model?

  4. What are the best resources that should be used to train the model?

  5. Which datasets I should use to train the model as well as documentation for the best practices etc?

  6. Currently I could find only one dataset available online for JS i.e. www.sri.inf.ethz.ch/js150. I don’t know the quality of it. If someone knows about it and can guide me if this can be used for training.

  7. I have a Macbook Pro M2 with 32 GB RAM. Will it be sufficient or will I need to go for a paid solution? And if paid solution which one is the cheap and best.

  8. Anything else that I should take care of?

Many thanks in advance and hoping for the support from the wonderful community.

2

Hi @bhupesh-sf ,

It’s great to see you diving into Machine Learning. I’m here to help you out with your project. Let’s break it down together.

  1. Choosing a Base Model:

I recommend starting with GPT-2. It’s a smaller model that’s easier to fine-tune and should run well on your Macbook.

  1. Base Model or Instruct-Ready Model:

Go with a base model first. We can always add instruction-based features later if you need them.

  1. Fine-Tuning the Model:

Use Hugging Face’s Transformers library. It’s user-friendly.
There are plenty of tutorials online that can guide you through the fine-tuning process step-by-step.

  1. Resources for Training:

Check out Hugging Face’s documentation and tutorials.
Online courses and community forums are also great places to learn and get help.

  1. Datasets for Training:

The JS150 dataset (www.sri.inf.ethz.ch/js150) is a good start.
You can also gather project data from your models, if any, to make the model even better.

  1. Macbook Pro M2 with 32 GB RAM:

Your Macbook should be fine for starting. If you need more power later, consider using cloud services like Google Colab or AWS. They offer free tiers that can be very handy.

Additional Tips:

Focus on one feature at a time to keep things manageable. Don’t hesitate to ask for feedback and support from the community. We’re all here to help each other.

Good luck with your project! Feel free to ask more questions as you go along.

1 Like
3

Many thanks for the detailed response and positive encouragement. If I have to start with 1st feature which is code completion, is there a specific pattern or steps to be followed for training?