Orchestration of Experts: The First-Principle Multi-Model System

We introduce our multi-model LLM building system to provide a high-quality AI system surpassing the capabilities of conventional single LLM systems.

At the heart of our innovation lies an LLM-based orchestrator trained to estimate the knowledge of underlying LLM experts. Given a list of underlying LLM experts encompassing generic and domain-specific models, our trained orchestrator predicts their performance for different queries without running their inference.

Evaluations on MMLU and GSM8K:

  • Using only open-source experts, the Leeroo model demonstrates superior accuracy by 5.27% over Mixtral
  • it achieves competitive results with GPT3.5 while reducing costs by 73.3% and being completely open source
  • Leeroo model attains competitive performance with GPT4, while sending 50% of queries to open-source experts
  • Leeroo Math 7B model achieved an accuracy of 84.77% in 5-shot setting, positioning it among the top performers in its class

More on our Leeroo HF organization.