How can I test of my PC/laptop is able to run new local models that get announced once in a while in The Batch newsletter? I am a GPU poor guy, but I possess a remote PC with a lot of RAM.
You can check for the parameter size of the model then make an estimate of whether your compute can run a model of such size.
1 Like
Check the Model’s Size: Look for the model’s parameter count (e.g., 7B, 13B) and its quantized versions (e.g., Q4_K_M), which are smaller and more memory-efficient.Use a Tool like Ollama: Install Ollama on your remote PC. It simplifies downloading and running models with a single command: ollama run <model_name>.Monitor RAM Usage: As the model runs, check your remote PC’s RAM usage. If the model fits in your RAM, it will work.Expect Slower Speed: Running on the CPU (using RAM) is much slower than a GPU, but you’ll still be able to test and use the model effectively.
1 Like
Did anybody come up with the exact formula? Downloading and monitoring every new model is not an option for me. I’d rather measure twice and pull once.
If you’re wondering which local LLMs you can run on your hardware, here’s a clear step-by-step guide to help you identify compatible models.
- Start by Understanding Your Hardware Specs
- Explore Which Models Fit Your Setup
- Use Tools that Detect What You Can Run
- Community Insights & Experimental Options
How?
Which?
Care to explain?
1 Like
Some of these things do not have an exact formula, but knowing your computer specs and also checking the model parameter size should give you a reliable estimate on whether you could run the model on your device. So just experiment for a bit and find the maximum parameter size that could run well on your device using a platform like ollama. Once you do that, you can know if a new announced model can be run locally by you.