Preference of LLMs API over custom models

Short Course Q&A ChatGPT Prompt Engineering for Developers
1

Hey Guys,
I wanted to know more about when should I prefer to use a LLM API over a custom model, which could either be trained from scratch, or could be a fine-tuned & pre-trained model. For instance, I can imagine various reasons, for when I might prefer a custom model:

  • One of the reasons has been highlighted in this thread. It would be great, if you guys can present your opinion about this as well.
  • Cost of inference; if I use a LLM API, I would have to pay it’s cost of usage, whereas, if I will use a custom model, it would be undoubtedly be on a much lower scale, when it comes to computation resources. Additionally, since my org will be the owner of the custom model, essentially there would be no middlemen such as “OpenAI”.
  • Data Security; using a LLM API pushes me to expose my data to the owner of LLM. It is not uncommon for any of us to see data breaches on a day-to-day basis, at present.
  • Larger Inference Time; undoubtedly a large model like ChatGPT will consume comparatively more time than a custom model (trained and optimised just for a single application) for inference.
  • I am not sure about this, but I believe that the performance of a custom model (single-purpose), will also be better than a LLM (multi-purpose), if our aim is to squeeze out the last drop of performance that exists.

Cheers,
Elemento

2

Interesting question. I would think it would depend on and the distinction of custom model (LLM trained in more specific data set?) and General model such as GPT. Your question would expand to camper GPT promoting and fine tuning mode. It would be interesting to make compare performance of different usage using labels Q&A pairs with some metrics (such as Blue Score).

3

Hey @takashisendo,
Thanks for the quick response.

The thing to note here is that it’s not just about the performance. Let’s say that I obtain a boost of 2-3% in performance, by using a LLM API, instead of a fine-tuned + pre-trained model. But will that 2-3% performance boost make up for the other cons, that I mentioned above. I can see that there are a lot of applications coming up nowadays, based on ChatGPT API, but does that count towards Responsible AI?

Cheers,
Elemento

4

You are right. Speed is easier to measure, but performance as a quality of answer also associated cost is said to be very different between GP3.5 API and GPT4 API (according to the BATCH May 10 2023). Application developer should think very carefully. No simple answer.

5

Hey @takashisendo,
Thanks a lot for your inputs. I do agree with what you stated, i.e., there is no simple answer to this question, and I guess the answer varies a lot from one application to another.

Cheers,
Elemento