We introduce our multi-model LLM building system to provide a high-quality AI system surpassing the capabilities of conventional single LLM systems.
At the heart of our innovation lies an LLM-based orchestrator trained to estimate the knowledge of underlying LLM experts. Given a list of underlying LLM experts encompassing generic and domain-specific models, our trained orchestrator predicts their performance for different queries without running their inference.
Evaluations on MMLU and GSM8K:
- Using only open-source experts, the Leeroo model demonstrates superior accuracy by 5.27% over Mixtral
- it achieves competitive results with GPT3.5 while reducing costs by 73.3% and being completely open source
- Leeroo model attains competitive performance with GPT4, while sending 50% of queries to open-source experts
- Leeroo Math 7B model achieved an accuracy of 84.77% in 5-shot setting, positioning it among the top performers in its class
More on our Leeroo HF organization.