Hi everyone!
If we want to use a non proprietary LLM that has sensitive specialized data to deliver counselling support (instead of information exchange), shall we link the LLM to a private database (to access the data) and use prompt engineering (to make it less informational but more of listening) OR shall we fine tune the LLM with the specialized data? Let’s say we can have access to local GPUs for fine tuning.
Guidance will be appreciated!
Hi @alex.a
I have some thoughts about your questions, but remember that in a professional environment, the right thing to do is to consult with your compliance team to ensure ethical procedures are followed.
So, when dealing with sensitive specialized data, it is crucial to prioritize data privacy and security. Both approaches, linking the Language Model (LLM) to a private database with prompt engineering and fine-tuning the LLM with the specialized data, have their pros and cons, and the choice depends on your specific requirements and constraints. Here are some considerations for each approach:
1. Linking LLM to a Private Database with Prompt Engineering:
Pros:
Data Privacy: By keeping the specialized data in a private database, you have more control over data access, reducing the risk of exposing sensitive information.
Prompt Engineering: You can craft custom prompts and responses that focus on delivering counseling support, making the interactions more personalized and empathetic.
Quick Implementation: Prompt engineering can be a faster approach to start delivering counseling support as you won’t need to fine-tune the entire model.
Cons:
Limited Customization: While prompt engineering allows some level of customization, it may still be limited compared to fine-tuning, where the model can learn directly from the specialized data.
Pretrained Bias: The LLM may retain some biases present in the pretraining data, which could potentially impact counseling interactions.
2. Fine-tuning the LLM with Specialized Data:
Pros:
Customization: Fine-tuning allows the LLM to learn from the specialized data directly, enabling a higher degree of customization for counseling support.
Better Understanding: Fine-tuning with specific data can help the model better understand the domain and context of counseling, potentially leading to more relevant and appropriate responses.
Reduced Bias: Fine-tuning can help mitigate potential biases by training the model on a more representative dataset.
Cons:
Data Privacy: Fine-tuning requires exposing the specialized data to the model, which may raise privacy concerns if not handled properly.
Resource-Intensive: Fine-tuning typically requires significant computational resources and time, even with access to local GPUs.
Model Retraining: If the specialized data changes or grows over time, you may need to periodically retrain the model to keep it up-to-date.
Recommendation:
Given the sensitive nature of the specialized data, the preference should be to prioritize data privacy and security. If it is possible to achieve the desired level of counseling support using prompt engineering while keeping the specialized data in a private database, that approach may be more appropriate. It allows you to maintain greater control over the data and reduces the risks associated with exposing sensitive information.
However, if prompt engineering proves to be insufficient or too limiting, and fine-tuning is necessary for a higher degree of customization, then it becomes crucial to implement robust privacy measures. This includes proper data anonymization, access controls, encryption, and ensuring compliance with data protection regulations.
Ultimately, the decision should be made with a thorough understanding of your specific use case, data sensitivity, available resources, and privacy requirements. If you have access to local GPUs and are considering fine-tuning, involve experts in data privacy and security to ensure compliance with best practices and regulations for handling sensitive data.
Of course i could be wrong in my statement, but i think this is a common approach in general manners.
I hope this help.
Best regards
Thank you Elirod!
Maybe we can solve the data privacy con of the fine tuning approach by using an open source model?
Yep!
This is a viable approach.
It really depends on your goals and desire outcomes.
Keep in mind that it is understandable that every model can become obsolete over time, because the data available for model retraining is changing over time too. That’s why the model must be adjusted as the results do not satisfy the needs of the business, for example .
It is in that context that the machine engineering team operates: working on the performance of the model to make its generalization more efficient.
Great. Thank you! GPUs may be needed 1) for LLMs to prepare/process the data and 2) for the eventual finetuning. How to estimate the size/configuration of GPUs needed in these two situations?
Estimating the size/configuration of GPUs needed for language model training and fine-tuning involves considering several factors such as model size, batch size, memory requirements, and training duration.
For most fine-tuning tasks with language models like GPT-3.5, you will likely need a high-end GPU with at least 16GB of VRAM. GPUs like NVIDIA GeForce RTX 3080 or NVIDIA A100 should be sufficient for many tasks. However, if you are working with extremely large models or datasets, you might need GPUs with even more VRAM, such as NVIDIA A6000 or NVIDIA A40.
The deep learning field is constantly evolving, and new hardware with improved capabilities might be available in the future. Therefore, always refer to the latest GPU specifications and consider the requirements of your specific task and dataset to make an informed decision. Additionally, using cloud-based GPU services can be a cost-effective option as it allows you to scale up or down based on your immediate needs.